Prefill vs Decode

Two phases, two completely different problems

You've seen prefill and decode mentioned throughout this series. Now it's time to understand them deeply — not just what they do, but why they behave so differently on the same GPU hardware. This distinction shapes every design decision in nano-vLLM and every optimisation in production inference engines.

The factory is compute-bound. The paint shop is memory-bandwidth bound. Same building, same power supply, completely different bottlenecks. This is exactly what happens with prefill and decode on a GPU.

Prefill — processing the input prompt

Prefill is the first phase of handling any request. When your prompt arrives — say, 500 tokens of text — the model processes all 500 tokens simultaneously in a single forward pass. All tokens go in together, all layers process them in parallel, and at the end the model has computed the Key and Value vectors for every input token and written them to the KV cache → Ch.03.

What makes prefill compute-bound?

To understand "compute-bound", you first need to understand the concept of arithmetic intensity — the ratio of math operations performed to memory bytes read. High arithmetic intensity = more computation per byte of data = the compute units (CUDA cores) are the bottleneck, not the memory bus.

During prefill, for each of the 500 input tokens, the model performs large matrix multiplications — multiplying the token's embedding vector against the model's weight matrices in every layer. Each weight matrix is read once but produces many multiply-accumulate operations. This is high arithmetic intensity. The GPU's thousands of floating-point units are all saturated. Adding more CUDA cores would make prefill faster. The memory bus is not the limit — the math units are.

Decode — generating tokens one at a time

After prefill, the model enters the decode phase — the autoregressive loop → Ch.01 where it generates one token per step. Each step processes only the single most-recently-generated token, reads the full KV cache for context, and samples the next token. This repeats until the sequence finishes.

Why decode is memory-bandwidth bound

Here's the key insight: to generate a single token, the GPU must read the entire model's weights from HBM → Ch.01 — all 7 billion parameters of a 7B model, or 14 GB at fp16. But it only uses those weights to perform computation for one token's worth of input. That's a tiny amount of math relative to the data movement. The arithmetic intensity is very low.

Think of it this way: you're reading 14 GB of data from HBM every decode step, but doing only a fraction of the math you'd do if you were processing a large batch of tokens. The memory bus is running flat out pumping weights from HBM to the compute cores, but the compute cores are only lightly loaded. Adding more CUDA cores would not help at all — they'd just sit idle waiting for data. The only thing that would help decode is faster HBM bandwidth.

TTFT and TPOT — the two latency metrics

Because prefill and decode behave so differently, inference engineers use two separate metrics to measure latency. Using just one metric hides the full picture.

TTFT — Time To First Token

TTFT (Time To First Token) measures how long after you send a request until you see the very first token of the response appear. This is entirely determined by prefill duration — the model must finish processing your entire prompt before it can generate token 1.

TTFT is what users feel as "responsiveness". A system with slow prefill feels sluggish even if it generates subsequent tokens quickly. A 5-second TTFT feels like the system is hanging, even if tokens stream out rapidly after that. For interactive applications, TTFT is often the most important metric.

TPOT — Time Per Output Token

TPOT (Time Per Output Token) measures the average time between each successive generated token after the first one. This is entirely determined by decode speed — specifically by HBM bandwidth, batch size, and model size.

TPOT is what users feel as "streaming speed" — how fast the text appears to flow after it starts. For long responses (essays, code, analyses), TPOT dominates the total wait time. A TPOT of 25ms means 40 tokens per second, which is roughly human reading speed — fast enough to feel natural.

Watch prefill and decode happen — step by step

Click the buttons below to see the two phases in action. Notice how prefill processes all input tokens in one shot, while decode advances one token at a time. Watch the KV cache fill up as each phase runs.

Chunked prefill — the best of both worlds

Prefill and decode have a tension: prefill is fast but blocks decode (no new tokens until prefill finishes), while decode is slow but generates tokens continuously. For a very long prompt, users stare at a blank screen during the entire prefill — terrible TTFT. Chunked prefill resolves this tension.

What is chunked prefill?

Instead of processing the entire prompt in one massive forward pass, chunked prefill splits the prompt into smaller chunks — say, 512 tokens at a time. After each chunk is processed, decode steps for existing sequences can be interleaved. The prompt is still fully processed before the new request generates any tokens, but other requests already in DECODING state keep generating throughout.

How the two phases appear in code

The clearest place to see the prefill/decode split is in the attention layer — the same function handles both phases, but dispatches to a completely different underlying kernel based on seq.is_prefill.

def forward(self, q, k, v, kv_cache, slot_mapping, block_table, is_prefill):
    # Step 1: Write this step's new K and V to the cache (both phases do this)
    store_kvcache_kernel[num_tokens,](k, v, kv_cache, slot_mapping, ...)

    if is_prefill:
        # ── PREFILL PATH ──────────────────────────────────────────────
        # All input tokens are in q, k, v as contiguous tensors.
        # flash_attn_varlen_func handles variable-length sequences
        # in a single batch, masking future tokens (causal mask).
        # This is compute-intensive: full attention over N tokens.
        out = flash_attn_varlen_func(
            q, k, v,
            cu_seqlens_q=cu_seqlens,   # cumulative sequence lengths for batching
            cu_seqlens_k=cu_seqlens,
            max_seqlen_q=max_seqlen,
            causal=True,               # don't attend to future tokens
        )
    else:
        # ── DECODE PATH ───────────────────────────────────────────────
        # Only the new token is in q. K and V for all prior tokens
        # are read from the paged KV cache via block_table.
        # flash_attn_with_kvcache handles non-contiguous block reads.
        # This is memory-bandwidth intensive: reads entire KV cache.
        out = flash_attn_with_kvcache(
            q,
            kv_cache[0],              # the full K cache tensor (all blocks)
            kv_cache[1],              # the full V cache tensor (all blocks)
            block_table=block_table,   # logical → physical block mapping
            cache_seqlens=seqlens,     # how many tokens are in cache per sequence
        )
    return out

def run(self, scheduler_output: SchedulerOutput):
    # Build a mixed batch: prefill sequences + decode sequences
    # Each sequence knows its phase via seq.is_prefill
    all_seqs = scheduler_output.prefill + scheduler_output.decode

    # Collect token IDs for the forward pass
    # Prefill sequences contribute ALL their tokens
    # Decode sequences contribute only their LATEST token
    input_ids = []
    for seq in all_seqs:
        if seq.is_prefill:
            input_ids.extend(seq.tokens)       # all N prompt tokens
        else:
            input_ids.append(seq.tokens[-1])   # just the last generated token

    # Single forward pass handles both prefill and decode sequences
    # The attention layer dispatches to the right kernel per sequence
    logits = self.model.forward(input_ids, ...)

Why understanding this changes how you think about LLM performance

Things beginners get wrong about the two phases

Quiz

Three questions on prefill, decode, and their bottlenecks. Wrong answers explain exactly where the reasoning went wrong.

What you now know