LLM Inference Fundamentals: Prefill, Decode, and the Memory-Compute Divide

LLM inference looks simple from the outside — feed in a prompt, get text back. Under the hood, the story is radically different from training. Training processes entire sequences in parallel with known labels. Inference must generate tokens one at a time, where each new token depends on every token before it. This autoregressive bottleneck shapes every aspect of how we design, optimize, and deploy inference systems.

This post is the foundational reference for understanding LLM inference performance. We will cover the two-phase execution model (prefill and decode), derive the arithmetic intensity of each phase from first principles, work through detailed memory calculations for the Llama 3 family (8B, 70B, and 405B), analyze how batching transforms the compute-to-memory ratio, discuss the serving challenges that arise at scale, and ground everything in real measured numbers on A100 and H100 hardware.

If you work on ML infrastructure, model serving, or simply want to understand why your chatbot sometimes feels slow, this is the post to start with.

---

Why Inference Differs from Training

Before diving into the mechanics of inference, it is worth understanding why inference is a fundamentally different computational problem from training — even though both execute the same neural network architecture.

Training: Full Parallelism

During training, we have the complete input-output sequence available upfront. A training step for a batch of sequences works like this:

1. Forward pass: Process all tokens in every sequence simultaneously. For a batch of $B$ sequences each of length $S$, the input to each transformer layer is a matrix of shape $[B \times S, d_{model}]$. The matrix multiplications are large and keep GPU compute units fully occupied.
2. Loss computation: Compare predicted tokens against ground-truth labels (which we already know) across all positions.
3. Backward pass: Compute gradients through the full computational graph. The backward pass involves the same large matrix multiplications as the forward pass, roughly doubling the total compute.
4. Optimizer step: Update all parameters using the computed gradients.

The critical insight is that training sees the entire sequence at once. There is no sequential dependency between generating token $t$ and token $t+1$ because we already have the ground truth for both. This means training is embarrassingly parallel across the sequence dimension, and the matrix multiplications are enormous — exactly what GPUs are designed for.

Inference: The Autoregressive Bottleneck

Inference is fundamentally sequential in the output dimension. To generate token $t+1$, the model must:

1. Have already generated token $t$ (and all tokens before it).
2. Run a full forward pass through all layers to produce a probability distribution over the vocabulary.
3. Sample or select token $t+1$ from that distribution.
4. Feed token $t+1$ back as input and repeat.

This means we cannot generate the 50th output token until we have generated the first 49. Each forward pass during generation processes only a single new token (or, in speculative decoding, a small handful). The matrix multiplications shrink from $[B \times S, d_{model}] \times [d_{model}, d_{ff}]$ during training to $[B \times 1, d_{model}] \times [d_{model}, d_{ff}]$ during generation — the input dimension collapses from $S$ (potentially thousands) down to $1$.

This is the autoregressive bottleneck. It means that during token generation, GPU compute units sit largely idle while the chip spends most of its time simply reading model weights from memory. The problem is not a lack of compute power; it is a lack of enough work to do per byte of data loaded.

The Roofline Perspective

The roofline model provides a clean framework for understanding this. A GPU has two ceilings: peak compute (measured in FLOPS) and peak memory bandwidth (measured in bytes/second). The ratio of these gives the ridge point — the arithmetic intensity (FLOP/byte) at which a workload transitions from memory-bound to compute-bound.

For an NVIDIA A100 SXM:

• Peak FP16 tensor core compute: $312$ TFLOPS
• Peak HBM bandwidth: $2.0$ TB/s
• Ridge point: $312 / 2.0 = 156$ FLOP/byte

For an NVIDIA H100 SXM:

• Peak FP16 tensor core compute: $989$ TFLOPS
• Peak HBM bandwidth: $3.35$ TB/s
• Ridge point: $989 / 3.35 = 295$ FLOP/byte

Any operation with arithmetic intensity below the ridge point is memory-bandwidth-bound. Any operation above it is compute-bound. As we will see, decode sits far below the ridge point, while prefill (with sufficiently long prompts) sits above it.

---

Prefill vs Decode: Two Phases, Two Bottlenecks

Every LLM inference request proceeds through two distinct phases that have opposite performance characteristics.

Phase 1: Prefill (Prompt Processing)

During prefill, all input prompt tokens are processed simultaneously in a single forward pass. If the prompt has $P$ tokens, the input to each transformer layer is a matrix of shape $[P, d_{model}]$. This flows through:

1. Self-attention: Query, key, and value projections are matrix multiplications of shape $[P, d_{model}] \times [d_{model}, d_{model}]$, producing $O(P \times d_{model}^2)$ FLOPs per projection (three projections plus the output projection). The attention score computation is $[P, d_{head}] \times [d_{head}, P]$ per head, which is $O(P^2 \times d_{head})$ per head.
2. Feed-forward network (FFN): Two (or three, for SwiGLU) matrix multiplications of shape $[P, d_{model}] \times [d_{model}, d_{ff}]$, producing $O(P \times d_{model} \times d_{ff})$ FLOPs each.

The key observation is that $P$ appears as a dimension of the input matrix. When $P = 512$ or $P = 2048$, these are large GEMMs (general matrix multiplications) with high arithmetic intensity. The GPU’s tensor cores are fully utilized.

Arithmetic intensity of prefill (FFN layer):

Consider a single FFN weight matrix of shape $[d_{model}, d_{ff}]$. The input is $[P, d_{model}]$.

• Bytes loaded: $d_{model} \times d_{ff} \times 2$ bytes (FP16 weights) $+ P \times d_{model} \times 2$ bytes (input activations). For large $d_{ff}$, the weight matrix dominates.
• FLOPs: $2 \times P \times d_{model} \times d_{ff}$ (multiply-accumulate).
• Arithmetic intensity: approximately $\frac{2 \times P \times d_{model} \times d_{ff}}{d_{model} \times d_{ff} \times 2} = P$ FLOP/byte.

For a prompt of $P = 512$ tokens, arithmetic intensity is $\sim 512$ FLOP/byte — well above the ridge point of both A100 ($156$) and H100 ($295$). Prefill is compute-bound.

In practice, prefill speed scales roughly linearly with prompt length (twice the prompt takes roughly twice as long, because there is twice the compute to do) until the prompt is so short that the operation falls below the ridge point.

Phase 2: Decode (Token Generation)

During decode, each forward pass processes a single new token. The input to each layer is a vector of shape $[1, d_{model}]$. This changes the computational profile dramatically:

1. Self-attention: The QKV projections become matrix-vector multiplications: $[1, d_{model}] \times [d_{model}, d_{model}]$. The attention computation reads the entire KV cache (all previous keys and values) but only produces one new query.
2. FFN: The multiplications become $[1, d_{model}] \times [d_{model}, d_{ff}]$ — again, matrix-vector products.

Arithmetic intensity of decode (FFN layer):

Same weight matrix $[d_{model}, d_{ff}]$, but now the input is $[1, d_{model}]$:

• Bytes loaded: $d_{model} \times d_{ff} \times 2$ bytes (weights still fully loaded).
• FLOPs: $2 \times 1 \times d_{model} \times d_{ff}$.
• Arithmetic intensity: $\frac{2 \times d_{model} \times d_{ff}}{d_{model} \times d_{ff} \times 2} = 1$ FLOP/byte.

An arithmetic intensity of $1$ FLOP/byte is catastrophically below the ridge point. The GPU’s tensor cores are idle more than $99\%$ of the time during single-request decode. Decode is memory-bandwidth-bound.

This table reveals the fundamental tragedy of LLM decode: you must read the entire model’s weights from memory for every single output token, but each token only does a trivial amount of arithmetic with those weights. The hardware is designed for workloads at $\gt 150$ FLOP/byte, and decode delivers $\sim 1$ FLOP/byte.

The KV Cache: Avoiding Redundant Computation

There is one crucial optimization that makes autoregressive generation practical at all: the KV cache. Without it, generating token $t$ would require recomputing attention over all $t-1$ previous tokens from scratch — meaning the cost of generating a sequence of length $T$ would be $O(T^2)$ in the sequence dimension (and $O(T^3)$ total when accounting for the $O(T)$ attention computation at each step).

The KV cache stores the key and value vectors computed during previous forward passes. When generating token $t$, we only need to:

1. Compute the new query, key, and value for the current token.
2. Append the new key and value to the cache.
3. Compute attention between the single new query and all $t$ cached keys/values.

This reduces the cost from $O(T^2)$ per step to $O(T)$ per step — a critical optimization. But it comes at the cost of memory: the KV cache grows linearly with sequence length and must be stored in GPU memory. As we will see, this memory pressure becomes the dominant constraint at scale.

---

Memory Math: Llama 3 Family

Let us work through the memory requirements for the three sizes in the Llama 3 family: 8B, 70B, and 405B. Understanding these numbers is essential for capacity planning.

Model Weights

The memory required for model weights depends on the number of parameters and the precision:

• FP16/BF16: 2 bytes per parameter
• INT8: 1 byte per parameter
• INT4: 0.5 bytes per parameter

Just the weights for Llama 3 405B in FP16 require over 810 GB — more than 10 A100-80GB GPUs just to hold the parameters with zero room for anything else. This is why quantization and model parallelism are not optional at this scale; they are mandatory.

KV Cache Memory

The KV cache is where things get interesting — and where most practitioners underestimate memory requirements. The KV cache stores key and value vectors for every layer, every head, and every token in every active request.

Per-token KV cache size:

$\text{KV per token} = 2 \times n_{layers} \times n_{kv\_heads} \times d_{head} \times \text{bytes\_per\_element}$

The factor of $2$ accounts for both keys and values. Note that Llama 3 uses Grouped Query Attention (GQA), where the number of KV heads ($n_{kv\_heads}$) is smaller than the number of query heads ($n_{q\_heads}$). This is specifically designed to reduce KV cache size.

Let us compute for each model:

Llama 3 8B: $n_{layers} = 32$, $n_{kv\_heads} = 8$ (GQA with 4:1 ratio from 32 query heads), $d_{head} = 128$

$\text{KV per token} = 2 \times 32 \times 8 \times 128 \times 2 = 131{,}072 \text{ bytes} = 0.125 \text{ MB}$

Llama 3 70B: $n_{layers} = 80$, $n_{kv\_heads} = 8$ (GQA with 8:1 ratio from 64 query heads), $d_{head} = 128$

$\text{KV per token} = 2 \times 80 \times 8 \times 128 \times 2 = 327{,}680 \text{ bytes} = 0.3125 \text{ MB}$

Llama 3 405B: $n_{layers} = 126$, $n_{kv\_heads} = 8$ (GQA with 16:1 ratio from 128 query heads), $d_{head} = 128$

$\text{KV per token} = 2 \times 126 \times 8 \times 128 \times 2 = 516{,}096 \text{ bytes} \approx 0.49 \text{ MB}$

Now let us scale this to realistic serving scenarios. A single request with a context length of $4{,}096$ tokens:

At one request with 4K context, the KV cache is modest. But serving is not about one request — it is about hundreds or thousands of concurrent requests.

With longer context lengths (32K, 128K, or the 128K context window that Llama 3 supports), the numbers become even more extreme. At 128K context with batch=32, Llama 3 8B would need $0.125 \times 131{,}072 \times 32 = 512$ GB of KV cache alone — an impossible number for a single GPU. This is why long-context serving requires aggressive KV cache compression techniques like quantization to INT8 or INT4, sliding window attention, or offloading to CPU memory.

Activation Memory

Activation memory is the working memory used during a forward pass to store intermediate results. Unlike model weights (which are constant) and the KV cache (which grows with sequence length), activation memory is proportional to the batch size and sequence length of the current forward pass.

During prefill, activation memory can be significant because the full prompt is processed at once. The dominant terms are:

• Attention scores: $O(B \times n_{heads} \times P^2)$ without FlashAttention, or $O(B \times n_{heads} \times P)$ with FlashAttention (which computes attention in tiles and does not materialize the full attention matrix).
• Layer intermediate outputs: $O(B \times P \times d_{ff})$ per layer.

For Llama 3 8B with $B=1$, $P=4096$: without FlashAttention, the attention matrix alone would be $32 \times 4096^2 \times 2$ bytes $= 1$ GB per layer, times 32 layers $= 32$ GB. With FlashAttention, this drops to a few hundred MB total. FlashAttention is not optional for long-context inference.

During decode, activation memory is negligible because we process only one token per step. The intermediate tensors are tiny vectors rather than large matrices.

Total Memory Budget

Let us put it all together for a concrete deployment scenario: Llama 3 70B on 2x A100-80GB (160 GB total) with FP16 weights, serving batch=32 at 4K context.

We are 29 GB over budget! The solution space includes:

1. Quantize weights to INT8: reduces weights from 141 GB to 70.6 GB, freeing 70 GB. Total becomes 119 GB — fits with headroom.
2. Quantize KV cache to INT8: reduces KV from 40 GB to 20 GB.
3. Reduce batch size: batch=16 halves KV cache to 20 GB, total becomes 169 GB… still tight.
4. Add more GPUs: 4x A100 gives 320 GB but increases communication overhead.

In practice, production deployments use a combination of INT8 or INT4 weight quantization, INT8 KV cache quantization, and enough GPUs to provide headroom for burst traffic. The memory budget is always the binding constraint.

---

Batch Size Effects: How Batching Changes Everything

Batching is the single most important lever in LLM inference performance. It transforms decode from a catastrophically underutilized operation into something approaching reasonable GPU utilization.

The Mechanics of Batched Decode

When we batch $B$ decode requests together, the forward pass processes $B$ tokens simultaneously. The FFN computation changes from a matrix-vector product to a matrix-matrix product:

• Unbatched: $[1, d_{model}] \times [d_{model}, d_{ff}]$ — matrix-vector
• Batched (B): $[B, d_{model}] \times [d_{model}, d_{ff}]$ — matrix-matrix (small GEMM)

The weight matrix $[d_{model}, d_{ff}]$ is loaded once from HBM regardless of batch size. But with batch size $B$, we perform $B$ times more FLOPs with that single read.

Arithmetic intensity of batched decode:

$\text{Arithmetic intensity} = \frac{2 \times B \times d_{model} \times d_{ff}}{d_{model} \times d_{ff} \times 2} = B \text{ FLOP/byte}$

At $B=1$: $1$ FLOP/byte (memory-bound). At $B=32$: $32$ FLOP/byte (still memory-bound on A100, but much better). At $B=156$: $156$ FLOP/byte (at the ridge point for A100 — balanced). At $B=295$: $295$ FLOP/byte (at the ridge point for H100 — balanced). At $B \gt 295$: Compute-bound on H100 — tensor cores become the bottleneck.

Throughput vs Latency Tradeoff

Increasing batch size improves throughput (total tokens generated per second across all requests) but degrades per-request latency (time for an individual token). Here is why:

• Throughput: Weight reads are amortized across $B$ requests. Total throughput scales roughly linearly with $B$ until we hit the compute ceiling or run out of memory.
• Latency: With larger batches, each forward pass takes longer because there is more compute to do. The KV cache also grows, making the attention computation slower. For a single user, their tokens come slower.

This is the fundamental tension in LLM serving: maximizing throughput for the provider vs minimizing latency for the user.

Several things to notice:

1. Throughput scales almost linearly from batch 1 to batch 64, then starts to plateau as compute utilization approaches the ceiling.
2. Per-request latency stays relatively flat up to batch 32 (the operation is still memory-bound, so adding work is “free”), then starts rising as we approach the compute ridge point.
3. KV cache memory scales linearly with batch size — at batch 256, it is 64 GB, consuming most of the A100’s memory.
4. The “sweet spot” for serving is typically in the range of batch 32-64, where throughput is high but latency is still acceptable and KV cache has not consumed all memory.

The Memory Wall

The maximum achievable batch size is constrained by GPU memory:

$B_{max} = \frac{\text{GPU memory} - \text{weights} - \text{overhead}}{\text{KV per request}}$

For Llama 3 8B on A100-80GB with FP16 weights and 4K context:

$B_{max} = \frac{80 - 16.1 - 4}{0.5} = \frac{59.9}{0.5} = 119$

For Llama 3 70B on 2x A100-80GB with FP16 weights and 4K context:

$B_{max} = \frac{160 - 141.2 - 6}{1.25} = \frac{12.8}{1.25} = 10$

Only 10 concurrent requests! This is why weight quantization is so important for larger models — it frees memory for KV cache, enabling larger batch sizes, which improves throughput. With INT8 weights (70.6 GB), the budget becomes:

$B_{max} = \frac{160 - 70.6 - 6}{1.25} = \frac{83.4}{1.25} = 66$

Quantizing weights from FP16 to INT8 increased max batch size from 10 to 66 — a $6.6\times$ improvement in maximum throughput, just from weight quantization. This is why quantization is listed as a serving optimization, not just a model compression technique.

---

The Serving Problem

Moving from a single inference request to a production serving system introduces a cascade of new challenges. Naive inference wastes enormous amounts of GPU resources, and the key innovations in LLM serving are all about closing this efficiency gap.

Why Naive Inference Wastes GPU

Consider a naive serving setup: a queue of requests, processed one at a time. Each request goes through prefill, then generates tokens until completion (or max length), then the next request starts.

The problems are severe:

1. Zero batching during decode: Each decode step reads the full model weights for a single token. On A100, this means $\lt 1\%$ compute utilization during generation — the most expensive hardware in the world is sitting idle.

2. Wasted prefill capacity: While one request is decoding, the GPU could be prefilling new requests (prefill is compute-bound and decode is memory-bound — they use different resources). But naive sequential processing cannot overlap them.

3. Variable request lengths: Requests have different prompt lengths and generate different numbers of output tokens. With static batching, the entire batch waits for the longest request to finish. If one request generates 10 tokens and another generates 500, the GPU is idle for $98\%$ of the time after the short request finishes.

4. Memory fragmentation: Different requests occupy different amounts of KV cache memory. As requests arrive and complete, the KV cache develops holes — allocated but unused memory regions that cannot be reclaimed for new requests.

Static Batching: The First Improvement

The simplest optimization is to batch multiple requests together: collect $B$ requests, prefill them all, then decode them all simultaneously. This improves throughput proportionally to the batch size (as discussed above).

But static batching has a critical flaw: all requests in a batch must start and end together. When the first request in a batch finishes generating, its slot sits empty until the entire batch completes. In practice, output lengths vary widely (some responses are 20 tokens, others are 2,000), so the average utilization of a static batch can be as low as $40\%$-$60\%$.

Continuous Batching: The Key Innovation

Continuous batching (also called iteration-level scheduling or inflight batching) solves this by making scheduling decisions at each decode step rather than at the batch level.

The core idea:

1. After each decode iteration, check if any requests in the batch have finished (hit EOS token or max length).
2. If so, immediately evict them and insert waiting requests from the queue.
3. The new requests go through prefill (which can be interleaved or chunked) and then join the decode batch.

This means the batch stays full continuously — as soon as one request finishes, another takes its place. Utilization jumps from $40\%$-$60\%$ to $85\%$-$95\%$ or higher.

The engineering challenge is managing the KV cache for a dynamically changing set of requests. When a request finishes and a new one starts, the new request’s KV cache must be allocated and the old one freed, without fragmenting the contiguous memory blocks that GPU kernels require.

Paged Attention and KV Cache Management

PagedAttention, introduced by the vLLM system, applies the operating system concept of virtual memory paging to KV cache management. Instead of allocating a contiguous block of memory for each request’s KV cache (which leads to internal fragmentation when requests have variable lengths), PagedAttention:

1. Divides KV cache memory into fixed-size blocks (analogous to memory pages).
2. Maintains a block table mapping each request’s logical KV positions to physical memory blocks.
3. Allocates blocks on demand as sequences grow, and frees them immediately when sequences complete.

This eliminates nearly all internal fragmentation and enables near-optimal memory utilization. In practice, PagedAttention increases the effective KV cache capacity by $2\times$-$4\times$ compared to contiguous allocation, enabling proportionally larger batch sizes.

The tradeoff is a small overhead from the indirection (looking up block tables during attention computation), but this is negligible compared to the throughput gains from better memory utilization.

Chunked Prefill and Prefill-Decode Overlap

Another important serving optimization is chunked prefill: instead of processing the entire prompt in a single forward pass (which can cause latency spikes for long prompts), split the prompt into chunks and process each chunk in a separate iteration. This allows:

1. Interleaving with decode: While processing a chunk of a new request’s prompt, the decode tokens from existing requests can be included in the same forward pass. This prevents prefill from stalling ongoing decode requests.
2. Bounded latency: No single iteration takes longer than a fixed time (determined by chunk size), making TTFT more predictable.
3. Better GPU utilization: Each iteration has a mix of prefill tokens (compute-heavy) and decode tokens (memory-heavy), which better utilizes both compute and memory bandwidth simultaneously.

The optimal chunk size balances prefill throughput (larger chunks = fewer overheads) against decode latency (larger chunks = longer iterations that delay decode tokens). Typical chunk sizes are 256-1024 tokens.

Request Scheduling and Preemption

In a production system with limited GPU memory, sometimes accepting a new request requires evicting (preempting) a partially completed request. The evicted request’s KV cache can be:

1. Dropped: The request is re-queued and must restart from scratch (re-prefill). Simple but wasteful.
2. Swapped to CPU memory: The KV cache is copied to CPU RAM (much larger but slower). When GPU memory becomes available, the cache is swapped back and decode resumes. This preserves work but adds swap latency.
3. Recomputed on demand: A hybrid approach where the prompt portion of the KV cache is recomputed (via prefill) but only on the portions needed for resuming generation.

Modern serving systems like vLLM, TensorRT-LLM, and SGLang implement sophisticated scheduling policies that balance throughput, latency, and fairness across concurrent requests.

---

Real Numbers: Latency and Throughput on A100 and H100

Theory is essential, but numbers are what matter for production decisions. The following benchmarks represent typical performance numbers for popular models on A100-80GB SXM and H100-80GB SXM GPUs, measured with optimized serving frameworks (vLLM, TensorRT-LLM) under realistic conditions. All numbers are approximate and vary with framework version, driver version, quantization calibration, and workload characteristics.

Hardware Comparison

For memory-bound operations (decode), the H100’s advantage is primarily from its $1.67\times$ higher memory bandwidth. For compute-bound operations (prefill), the advantage is $3.17\times$ (FP16) or even higher with FP8.

Llama 3 8B: Single GPU Performance

Key observations for the 8B model:

• At batch=1, throughput is determined almost entirely by memory bandwidth. The H100’s $1.67\times$ bandwidth advantage translates to a $1.7\times$ speedup in decode.
• INT8 quantization on A100 provides a $\sim 1.2\times$ throughput improvement at small batch sizes (from reduced memory reads) and $\sim 1.2\times$ at large batch sizes (from fitting more requests).
• FP8 on H100 provides the best overall performance: faster compute for prefill and reduced memory footprint for larger batches.
• The maximum batch size approximately doubles with quantization (halving the weight memory frees room for more KV cache).

Llama 3 70B: Multi-GPU Performance

Key observations for the 70B model:

• Tensor parallelism (TP) is essential. On 2x A100 with FP16 weights, max batch is only ~10 — barely viable for production. The memory is almost entirely consumed by weights.
• 4x H100 with FP8 is the sweet spot for many production deployments: good single-request latency (~3.5 ms/tok ITL), strong throughput at batch 32 (7,200 tok/s), and enough memory for 120 concurrent requests.
• 8x H100 provides diminishing returns per GPU but offers the lowest absolute latency (2.2 ms/tok ITL) and highest throughput for latency-sensitive applications.
• TTFT varies dramatically with tensor parallelism: more GPUs = faster prefill (compute splits across GPUs, though NVLink communication adds overhead).

Llama 3 405B: The Scale Challenge

At 405B parameters, the operational reality is stark:

• Minimum 8 H100s (with FP8) just to fit the model in memory with room for a modest batch. With FP16, you need 16+ GPUs.
• Pipeline parallelism (PP) becomes necessary alongside tensor parallelism when model weights exceed the combined memory of a single NVLink domain (typically 8 GPUs). PP adds pipeline bubbles that reduce efficiency by $10\%$-$20\%$.
• Cost per token is extremely high. At $\sim 2,000$ tok/s on 16x H100 with batch=16, the cost is roughly $8\times$ higher per token compared to the 70B model on 4x H100 at batch=32.
• For most production use cases, the 70B model with good quantization provides a much better cost-performance tradeoff than the 405B model.

Decode Throughput Scaling Curves

To illustrate how throughput scales with batch size on real hardware, here are measured curves for Llama 3 8B:

The scaling pattern is consistent: near-linear throughput growth with batch size in the memory-bound regime, followed by a plateau as compute becomes the bottleneck. The H100’s higher memory bandwidth pushes the transition point to a higher batch size, and its higher compute ceiling raises the plateau.

---

Putting It All Together: The Mental Model

Let us synthesize everything into a coherent mental model for LLM inference performance.

The Three Resources

LLM inference performance is determined by three scarce resources:

1. Compute (FLOPS): Tensor core throughput. Limits prefill speed and decode throughput at high batch sizes.
2. Memory bandwidth (GB/s): HBM read throughput. Limits single-request decode speed and decode throughput at low-to-moderate batch sizes.
3. Memory capacity (GB): Total HBM. Limits model size, maximum batch size (via KV cache), and maximum context length.

At any given moment, one of these is the binding constraint. The art of inference optimization is shifting the bottleneck to a different resource (typically from memory bandwidth to compute, via batching) or expanding the binding resource (more GPUs, faster memory, quantization to reduce capacity pressure).

Decision Framework

When evaluating an inference deployment, ask these questions in order:

1. Can the model fit in memory? Calculate: weights + KV cache (at target batch size and context length) + overhead. If this exceeds available GPU memory, you need more GPUs, quantization, or a smaller model.

2. What is the decode bottleneck? Calculate arithmetic intensity at your target batch size. If $B \lt$ ridge point, you are memory-bandwidth-bound: faster memory or more batching helps; more compute does not. If $B \gt$ ridge point, you are compute-bound: faster compute helps; more memory bandwidth does not.

3. What limits batch size? Usually KV cache memory. Calculate $B_{max}$ as shown above. If $B_{max}$ is below the ridge point, you will never fully utilize compute — consider KV cache quantization, GQA, or more memory to increase $B_{max}$.

4. What are the latency requirements? TTFT is bounded by prefill time (proportional to prompt length, inversely proportional to compute throughput). ITL is bounded by decode step time (inversely proportional to memory bandwidth at low batch, or compute at high batch). If latency requirements are strict, you may need to sacrifice throughput (smaller batch) or add more GPUs (split work).

The Optimization Stack

Each optimization targets a specific bottleneck:

The Cost Equation

Ultimately, inference is about cost per token. The cost equation is:

$\text{Cost per token} = \frac{\text{GPU cost per second}}{\text{Throughput (tok/s)}}$

Every optimization that increases throughput (without proportionally increasing GPU cost) reduces cost per token. This is why batching is so powerful: it increases throughput by $10$-$30\times$ with zero additional hardware cost.

For a concrete example: Llama 3 70B on 4x H100 (cost: \sim\16$/hour in cloud pricing):

• Batch=1: $\sim 286$ tok/s, cost = \16 / 3600 / 286 \approx $0.0000155$per token ($$15.5$ per million tokens)
• Batch=32: $\sim 7{,}200$ tok/s, cost = \16 / 3600 / 7200 \approx $0.00000062$per token ($$0.62$ per million tokens)

Batching reduced cost by $25\times$. This is why API providers can offer LLM inference at prices that seem impossibly low — they are batching aggressively across thousands of concurrent requests.

---

Conclusion

LLM inference is defined by a fundamental asymmetry: prefill is compute-bound (large matrix multiplications that saturate tensor cores), while decode is memory-bandwidth-bound (full model weight reads for each single generated token). This asymmetry, rooted in the autoregressive nature of language generation, drives every design decision in the inference stack.

The key quantitative relationships to internalize:

• Arithmetic intensity during decode = batch size (in FLOP/byte). At batch=1, utilization is below $1\%$. The ridge point for A100 is $\sim 156$, for H100 is $\sim 295$.
• KV cache scales as $2 \times n_{layers} \times n_{kv\_heads} \times d_{head} \times \text{seq\_len} \times \text{batch\_size}$. It dominates memory at production batch sizes.
• Maximum batch size is memory-limited, and quantization’s primary serving benefit is freeing memory for larger batches.
• Throughput scales linearly with batch size until compute saturation, then plateaus.

The serving innovations — continuous batching, PagedAttention, chunked prefill, KV cache quantization — all serve a single goal: maximize the number of concurrent requests the GPU can process, thereby amortizing the fixed cost of reading model weights across more useful output tokens.

If you take away one thing from this post: LLM decode performance is not about making the GPU compute faster. It is about giving the GPU more useful work to do with each byte it reads from memory. Every optimization in the inference stack — from batching to quantization to paged memory management — is ultimately in service of this principle.