learn deep. ship safe.Practice testsFollow ByteMunSubscribe

Concept

LLM inference and cost

Serving a model's answers: latency, throughput, batching, GPUs and cost per token.

LLM inference and cost explained

LLM inference is the work of running a trained language model to produce an answer. The model first reads the whole prompt in one pass (prefill), then generates the reply one token at a time (decode), each new token depending on all the ones before it. Speed and cost are driven mostly by how many tokens go in and come out, the size of the model, and how well the server batches requests.

Prefill and decode

  • Prefill processes all input tokens in parallel. Long prompts make this phase heavier and raise the time to first token.
  • Decode produces output tokens one after another. Its speed, tokens per second, sets how long a long answer takes.
  • The KV cache stores intermediate results for tokens already processed, so each new token does not recompute the whole sequence. It also uses a lot of GPU memory, which limits how many requests fit at once.

What makes inference faster or cheaper

  • Batching: serving many requests together keeps the GPU busy and lowers cost per token.
  • Quantization: storing weights in fewer bits so a model needs less memory and runs faster, usually with a small quality cost.
  • Prompt caching: reusing the processed form of a long, repeated prefix such as a system prompt or document.
  • A smaller model for simple steps (classification, routing, extraction), keeping the large one for hard reasoning.
  • Streaming: sending tokens as they are generated so the user starts reading sooner, even if the total time is the same.

How LLM cost works

Hosted APIs bill by tokens, usually with different rates for input and output tokens, and output tokens typically cost more because decode is the slow phase. Cost per request is roughly input tokens times the input rate plus output tokens times the output rate, so long system prompts, large retrieved contexts and verbose answers all add up. Measure tokens per request before optimising anything else.

Common mistakes

  • Sending the whole document when retrieval of the relevant parts would do.
  • Using the largest model for every step of a pipeline.
  • Judging speed by average latency alone instead of time to first token and the slowest requests.

What interviewers ask about LLM inference

  • "Why is the first token slower than the rest?" Prefill has to process the whole prompt first.
  • "How would you cut cost by half without losing quality?" Shorter prompts, caching, routing easy requests to a smaller model, and capping output length.
  • "What is the KV cache and why does it limit concurrency?" It saves recomputation but its memory grows with every token of every active request.
Concept map
Part of AI system designLLM inference and cost

What should you learn before LLM inference and cost?

  1. Deep learningMachine learning with many-layered neural networks.
  2. Tokens and tokenizationCost and latency are counted in tokens.
  3. TransformersThe attention-based neural network architecture behind modern language models.
  4. Large language models (LLMs)Transformer models trained on large amounts of text to predict the next token.

Connections to other concepts

Read and watch: LLM inference and cost explained

Practise LLM inference and cost

All topics