nano-vLLM Architecture

Why architecture matters

Before touching any concept — KV cache, PagedAttention, scheduling — you need to know where those things live in the codebase. Architecture is the map. Without it, you're reading code in the dark.

This separation is not cosmetic — it's the key design decision that makes the entire engine fast. The CPU can make scheduling decisions without blocking the GPU, and the GPU can run compute without waiting for Python-level bookkeeping.

Every file, explained

The entire nano-vLLM codebase is ~1,200 lines across 11 files. Click any file below to see what it does, why it exists, and how it fits into the bigger picture.

CPU control plane vs GPU data plane

The single most important architectural decision in nano-vLLM — and in production vLLM — is the strict separation between what runs on the CPU and what runs on the GPU. Understanding this division makes everything else in the codebase make sense.

What "metadata" actually means

When we say the CPU only manages metadata, we mean small, cheap Python data structures — not tensors:

Four layers, one request

When you call llm.generate(), your request passes through four distinct architectural layers. Each layer has exactly one responsibility and hands off to the next. Click each layer to understand its role.

One request's journey through the architecture

Let's trace a single generate() call through every layer from the moment you call it to the moment you receive text back. Each hand-off between layers is where the CPU/GPU boundary matters most.

The engine loop in code

The engine's step() method is the core of the architecture — it calls the scheduler, runs the model, and updates sequences. Here it is, annotated:

def step(self) -> list[SequenceOutput]:
    # 1. Ask the CPU scheduler what to run this step.
    #    Returns prefill sequences + decode sequences as a batch.
    scheduler_output = self.scheduler.schedule()

    # 2. If nothing to run (all requests waiting for memory), return empty.
    if not scheduler_output.has_work():
        return []

    # 3. Hand the batch to the GPU model runner.
    #    This is where the transformer forward pass happens.
    #    Returns one sampled token ID per sequence in the batch.
    sampled_tokens = self.model_runner.run(scheduler_output)

    # 4. Update each sequence with its new token.
    #    The scheduler handles state transitions (prefill → decode,
    #    decode → finished) and block deallocation on completion.
    outputs = self.scheduler.update(sampled_tokens)

    return outputs  # list of (sequence_id, token_id, is_finished)

The Sequence object — the unit of work

Every request is represented as a Sequence object. It's the single piece of state that flows between the CPU control plane and GPU data plane:

class Sequence:
    def __init__(self, prompt_tokens: list[int], params: SamplingParams):
        self.tokens = prompt_tokens    # all token IDs so far (grows with each decode step)
        self.params = params           # temperature, top_k, max_tokens etc.

        # CPU metadata — managed by the block manager
        self.block_table: list[int] = []   # [physical_block_id, ...] for this sequence

        # State machine — managed by the scheduler
        self.status = SequenceStatus.WAITING   # WAITING → PREFILL → DECODING → FINISHED
        self.is_prefill = True               # True until first token is generated

        # Output accumulator
        self.output_tokens: list[int] = []  # generated token IDs (not including prompt)

What this architecture enables

The CPU/GPU separation and the four-layer design aren't just clean code — they're what makes the performance features in later chapters possible.

Things beginners get wrong about LLM architecture

Quiz

Three questions on the architecture. Wrong answers tell you exactly why they're wrong.

What you now know