Retrieval, Vector Search & Production RAG

1. Decoder-only anatomy, priced per component

Interviewers for platform and architect roles rarely want the training math. They want to know whether you can map each architectural component to a runtime cost, because that mapping is what makes serving decisions rational. A modern chat model is a decoder-only transformer: it converts text into tokens, tokens into vectors, pushes those vectors through a stack of identical blocks, and emits a probability distribution over the next token.

Tokenization is byte-pair encoding (BPE) or a byte-level variant: a learned merge table that greedily compresses frequent byte sequences into single vocabulary entries (vocabularies today run 32K–256K). Everything downstream — context limits, latency, and your invoice — is denominated in tokens, and tokenizers are not interchangeable: code, numbers, and non-English text can tokenize 1.5–4× longer than English prose, which silently changes both cost models and cross-engine benchmark comparisons. Embeddings map each token ID to a d_model-dimensional vector; position is injected not by adding position vectors but by RoPE (rotary position embeddings), which rotates query/key vectors by an angle proportional to position — a detail that matters later because it is the hook for context extension.

Each block applies pre-normalization (RMSNorm), self-attention, a residual add, another norm, then a gated MLP (SwiGLU, expanding to roughly 3–4× d_model), and a second residual add. Two facts earn senior credit: about two-thirds of parameters live in the MLPs, not attention — so weight memory and decode bandwidth are mostly an MLP bill; and the residual stream means each block edits a shared representation rather than replacing it, which is why layers can be quantized or even skipped with graceful rather than catastrophic degradation. Attention computes, for every position, a similarity score against every prior position — an n × n score matrix, which is the O(n²) term everyone cites. Crucially, that quadratic cost is paid during prompt processing; once past keys/values are cached, each new token attends to n cached entries — linear per step.

MHA → MQA → GQA → MLA: shrinking the cache, not the compute

The attention variants exist for one dominant reason: the KV cache (next section) scales with the number of key/value heads. Multi-query attention (MQA) shares one K/V head across all query heads; grouped-query attention (GQA) shares K/V among groups — Llama-3-70B uses 8 KV heads against 64 query heads, an 8× cache reduction; multi-head latent attention (MLA, introduced by DeepSeek-V2) caches a low-rank latent compression of K/V, cutting cache size by an order of magnitude at the price of extra projection compute.

2. KV cache and the two-phase request

Every autoregressive request has two phases with opposite performance characters. Prefill processes all prompt tokens in parallel — big matrix multiplies, high arithmetic intensity, compute-bound — and its duration is essentially your time-to-first-token (TTFT). Decode generates one token per forward pass; each step must stream the model weights and the growing KV cache through the GPU's memory system to do comparatively little math, so it is memory-bandwidth-bound, and it sets time-per-output-token (TPOT).

sequenceDiagram
  participant C as "Client"
  participant R as "API gateway"
  participant S as "Engine scheduler"
  participant G as "GPU worker"
  C->>R: submit prompt
  R->>S: enqueue request
  S->>G: admit and allocate KV blocks
  G->>G: prefill all prompt tokens in one pass
  G-->>C: first token streamed
  loop one token per step until stop
    G->>G: read weights plus KV cache
    G-->>C: stream next token
  end
  G->>S: release KV blocks
    

The KV cache stores every layer's keys and values for every token so decode never recomputes them. Memorize the formula and one worked example:

kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes

Llama-3-70B (80 layers, 8 KV heads, head_dim 128, FP16):
  2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 320 KB per token
  → 8K-token session  ≈ 2.6 GB    → 32K ≈ 10.5 GB    → 128K ≈ 42 GB per sequence

Set that against hardware: FP16 weights for a 70B model are ~140 GB, so an 8×H100 node (640 GB HBM) has roughly 400+ GB left for KV after weights and activation workspace — call it ~1.2M cached tokens, or about 300 concurrent 4K-context sessions, but only nine concurrent 128K sessions. This single calculation explains why long context is an economics problem before it is a quality problem, and why GQA/MLA and KV quantization exist at all.

The roofline argument is worth reciting. At batch 1, decoding one token reads every weight byte to perform ~2 FLOPs per parameter — arithmetic intensity near 1 FLOP/byte, while an H100 needs roughly 295 FLOPs per byte moved (~989 dense BF16 TFLOPS over ~3.35 TB/s HBM3) to saturate. Hence the batch-1 ceiling: an 8B model in FP16 (~16 GB) on one H100 cannot exceed ~3,350/16 ≈ 200 tokens/s no matter how clever the kernels, and measured numbers sit below that. Batching multiplies useful FLOPs per weight byte read, which is why every serving optimization ultimately serves one goal: keep many sequences in flight per weight pass. Prefill math is equally quotable: a 2,000-token prompt on that 8B model needs ~2 × 8e9 × 2000 = 32 TFLOPs, so at ~50% utilization TTFT has a floor around 65 ms — before queueing, which dominates in practice.

3. Continuous batching and PagedAttention

Naive serving batches requests statically: collect N prompts, run them together, return when the longest finishes. Two pathologies follow — the batch runs at the speed of its slowest member while finished slots idle, and arrivals wait for the next batch to form. Orca introduced iteration-level scheduling — continuous batching — where the scheduler recomposes the batch at every decode step: finished sequences leave immediately, queued requests join mid-flight. This alone yields multi-fold throughput gains over static batching and is table stakes in every modern engine.

flowchart TD
  Q["Incoming request queue"] --> A["Admission control checks free KV blocks"]
  A -->|"blocks available"| B["Join running batch at next step"]
  A -->|"pool exhausted"| P["Queue, or preempt a sequence by swap or recompute"]
  B --> F["Single forward pass for whole batch"]
  F --> D{"Sequence hit stop token or max length?"}
  D -->|"yes"| E["Stream final token and free KV pages"]
  D -->|"no"| B
  E --> A
    

Continuous batching made KV allocation the new bottleneck. Pre-PagedAttention engines reserved contiguous KV memory for each request's maximum possible length; measured waste from internal fragmentation and over-reservation ran 60–80% of KV memory. PagedAttention (the vLLM paper) applies virtual-memory thinking: KV lives in fixed-size blocks (e.g., 16 tokens), each sequence holds a page table, and blocks are allocated on demand — waste drops to under ~4%, which converts directly into batch size and therefore throughput. Paging also enables prefix sharing with copy-on-write: a thousand requests carrying the same 2K-token system prompt can reference one physical copy of its KV blocks. SGLang generalized this into RadixAttention, an automatic prefix-cache tree, which is why it shines on agentic workloads that re-send conversation prefixes on every step (Chapter 05 depends on this).

• Chunked prefill — split large prompt prefills into slices interleaved with decode steps, so one 100K-token upload does not spike everyone else's TPOT.
• Preemption — when the KV pool exhausts, engines evict a sequence (recompute later, or swap to CPU). Watch this metric: preemptions are the canary for KV pressure.
• Prefill–decode disaggregation — run prefill and decode on separate GPU pools and ship KV between them; removes phase interference at the cost of a KV transfer path. An emerging default for large deployments.
• Goodput, not throughput — the number that matters is tokens/s delivered while meeting TTFT/TPOT SLOs, not peak tokens/s at unbounded latency.

4. The quantization ladder

Quantization is the highest-leverage cost knob after batching, because decode speed is proportional to bytes moved. The ladder runs from BF16 (training-native baseline) down to 4-bit weights, and the senior move is knowing which rung serves which regime: weight-only quantization accelerates the memory-bound low-batch regime; weight-and-activation formats (FP8/INT8) exploit faster tensor cores and win in the compute-bound high-batch regime, where weight-only 4-bit can actually lose to FP8 because of per-step dequantization overhead.

The quality caveat is where candidates get filtered. Aggregate benchmarks (MMLU-style) routinely show <1% degradation for well-executed 4-bit quantization of large models, while narrow capabilities — math, code generation, low-resource languages, strict instruction-following at long context — degrade first and disproportionately. GPTQ minimizes layer-wise reconstruction error using second-order information; AWQ instead protects the ~1% of weight channels with the largest activation magnitudes. Both are calibration-dependent: quantize with generic web-text calibration data and evaluate on your legal-summarization traffic, and you may see regressions the model card never showed. The rule: the quantization decision is an evaluation decision — your own task harness (Chapter 06) gates every rung change.

flowchart TD
  S["Start at BF16 with a task eval harness"] --> Q1{"Bottleneck today?"}
  Q1 -->|"memory capacity or low-batch latency"| W4["Weight-only 4-bit AWQ or GPTQ"]
  Q1 -->|"throughput at high batch on Hopper"| F8["FP8 weights and activations"]
  Q1 -->|"KV capacity limits concurrency"| KV["FP8 KV cache first"]
  W4 --> E["Re-run task evals plus latency benchmark"]
  F8 --> E
  KV --> E
  E --> G{"Quality within agreed budget on your slices?"}
  G -->|"yes"| SHIP["Ship, monitor drift and complaints"]
  G -->|"no"| ROLL["Step back up one rung or change calibration set"]
    

5. Faster decode: speculation, MoE, and long context

Three techniques dominate the “make decode cheaper” conversation, and each has a regime where it backfires. Speculative decoding (Leviathan et al., Chen et al.) uses a cheap drafter to propose k tokens, which the target model verifies in a single parallel pass; rejection sampling guarantees the output distribution is exactly the target model's. Because verification prices like a tiny prefill, you convert several bandwidth-bound steps into one compute-heavier step — a 1.5–3× TPOT win when acceptance rates are high (drafter matches the target's style; predictable text like code) and batch is small. At high batch the GPU is already compute-saturated, so speculation adds work and can reduce throughput; engines increasingly disable it dynamically under load. EAGLE and Medusa-style approaches replace the separate drafter with lightweight heads on the target model itself, removing the two-model operational burden.

flowchart LR
  DR["Drafter proposes k tokens"] --> V["Target verifies all k in one parallel pass"]
  V --> AC{"How much of the draft matched?"}
  AC -->|"all k accepted"| N["Emit k plus one bonus token"]
  AC -->|"partial"| PR["Emit accepted prefix plus one corrected token"]
  N --> DR
  PR --> DR
    

Mixture-of-experts models decouple parameters from per-token compute: Mixtral 8×7B holds ~47B parameters but routes each token through ~13B. The serving implications are the interview substance: all experts must be resident, so memory is priced at total parameters while compute is priced at active parameters; at batch 1 you read only the routed experts (bandwidth win), but as batch grows nearly every expert is activated by someone, so weight-read amortization converges back toward dense behavior; and multi-GPU MoE adds expert-parallel all-to-all communication plus load-balancing pathologies when routing concentrates on hot experts. MoE is a hardware-utilization bet, not a free lunch.

Long context is bounded by two costs you now own: KV memory growing linearly (Section 2's formula) and prefill compute growing quadratically-ish in practice. Models extend beyond trained context by rescaling RoPE — position interpolation, NTK-aware scaling, and YaRN — which stretches rotary frequencies so unseen positions land in familiar ranges; quality at extended lengths must be verified with needle-and-haystack and task-level evals, since "supports 128K" often means "does not crash at 128K." Operationally, pair long context with FP8 KV, chunked prefill, and prefix caching — and remember that the cheapest long-context token is the one you retrieved instead (Chapter 04's argument).

6. Serving engines: vLLM, TGI, TensorRT-LLM, SGLang

All four mainstream engines now implement continuous batching, paged KV, quantized formats, and OpenAI-compatible APIs, so the differentiation is operational: kernel pedigree, model-coverage velocity, prefix-caching sophistication, and how much build engineering they demand.

Interview framing that lands: the engine choice is reversible (they share API shapes); the benchmark methodology is what protects the decision. Comparing engines on tokens/s with different tokenizers, different default sampling, or unmatched quantization is the classic self-deception — normalize the workload first (Section 9).

7. GPU economics and build-versus-buy

Cost per million tokens is the unit that makes inference decisions comparable across managed APIs and self-hosting. The formula is trivial — GPU-hour cost divided by sustained tokens per hour — and every input is a place candidates go wrong: they use peak benchmark throughput instead of SLO-compliant goodput, and they assume 100% utilization when real diurnal traffic delivers 20–40%.

8. Serving on AWS and GCP

Both clouds offer the same three altitudes — fully managed per-token APIs, managed endpoints you configure, and raw GPUs you orchestrate — and interviewers for cloud-aligned roles expect you to pick an altitude from workload characteristics, then name concrete services fluently.

flowchart TD
  R["New GenAI workload"] --> Q1{"Frontier-model quality required?"}
  Q1 -->|"yes"| M["Managed API on Bedrock or Vertex AI"]
  Q1 -->|"open weights pass evals"| Q2{"Steady traffic at high utilization?"}
  Q2 -->|"no, spiky or unproven"| M2["Managed endpoints or serverless GPUs, pay as you go"]
  Q2 -->|"yes"| Q3{"Team ready to own GPU operations?"}
  Q3 -->|"yes"| SH["Self-host vLLM on EKS or GKE"]
  Q3 -->|"no"| PT["Provisioned throughput or managed endpoints"]
  M --> RT["Re-evaluate quarterly as volume and models shift"]
  SH --> RT
    

9. TTFT, TPOT, throughput — benchmarking without self-deception

Three metrics, three different masters. TTFT (time to first token) is queueing plus prefill — it gates perceived responsiveness in chat and agent loops. TPOT (time per output token, a.k.a. inter-token latency) is the decode rhythm — it gates streaming readability and total completion time. Throughput (aggregate tokens/s per replica) is the cost side. They trade against each other through batch depth: deeper batches raise throughput and TPOT together, and a benchmark that reports one without the others is marketing.

1. Fix the workloadSample real prompt/output length distributions — a 6K-in / 300-out RAG trace behaves nothing like ShareGPT chat. Same tokenizer accounting across engines.
2. Sweep loadRamp concurrency or request rate stepwise; at each level record TTFT and TPOT percentiles plus aggregate tokens/s.
3. Find the kneeIdentify where p95 TTFT or TPOT breaches SLO; the sustained rate just below it is your capacity number.
4. Price itCost per million tokens at the knee — not at peak throughput — feeds the build-vs-buy model of Section 7.
5. Re-run on changeNew model, quant rung, engine version, or context policy re-runs the sweep; keep results versioned like eval results.

The dishonesty patterns to name in an interview: benchmarking at batch 1 and deploying at batch 64; warm prefix caches flattering TTFT for prompts your real traffic never repeats; comparing engines with different tokenizers so “tokens/s” measures the tokenizer, not the engine; quoting mean latency where tail latency gates the product; and quoting a managed API's demo-time latency without measuring its variance under your regional, peak-hour traffic. A candidate who volunteers these earns instant credibility.

Interview playbook

For any inference-performance question, run BUDGET — size the physics before naming products:

1. B — Bytes: weights at the chosen precision, plus KV per token × context × concurrency. Does it fit, and on how many GPUs?
2. U — Utilization regime: is the phase compute-bound (prefill, high batch) or bandwidth-bound (decode, low batch)? The regime picks the optimization.
3. D — Demand shape: traffic pattern, prompt/output length distribution, prefix reuse, latency SLOs per product surface.
4. G — Gains ladder: continuous batching → paged KV/prefix cache → quantization rung → speculative decoding → disaggregation, each gated by evals.
5. E — Economics: cost per million tokens at SLO-compliant load; compare managed API vs provisioned vs self-host at the actual utilization.
6. T — Test honestly: load-swept percentiles on a realistic trace, re-run on every change.

Common traps

• Citing O(n²) attention as why long context is expensive, instead of KV memory pressure and prefill interference.
• Treating quantization as free compression — no calibration story, no task-slice evals, no rollback rung.
• Recommending speculative decoding for a saturated high-batch cluster where it burns the compute headroom it needs.
• Comparing self-hosting at 100% assumed utilization against a managed API's list price.
• Benchmark numbers without percentiles, load levels, or the prompt-length distribution attached.
• Naming vLLM features without being able to explain what PagedAttention actually fixed (fragmentation and over-reservation of KV memory).

Question bank

These are the questions senior GenAI platform and architect loops actually ask about inference. Practice deriving, not reciting.

Proof artifact: an honest inference benchmark and cost model

Build a small, reproducible study you can defend line by line: one open model, one engine, a load harness, and a one-page decision memo. This artifact backs answers across Sections 2–9 and pairs with the evaluation harness of Chapter 06.

Steps

1. Deploy an 8B-class instruct model with vLLM on a single cloud GPU (an L4 or A10G-class instance keeps example cost low); record exact model, engine version, dtype, and max context.
2. Build a load generator that replays a realistic trace: sampled prompt lengths (include a long-prompt slice), realistic output lengths, streaming enabled, fixed random seed.
3. Sweep concurrency (e.g., 1, 2, 4, 8, 16, 32, 64) and record p50/p95/p99 TTFT and TPOT plus aggregate tokens/s at each level; chart the saturation curve and mark the SLO knee.
4. Compute cost per million output tokens at the knee from the instance's hourly price; then redo the number at 25% assumed utilization and compare with two managed-API list prices as reference points.
5. Quantize to AWQ (or FP8 if the GPU supports it), re-run both the load sweep and a ~100-item task eval with slices; record the quality delta next to the speed delta.

Deliberate failures

• Drive KV exhaustion with many long-context sessions; capture preemption/queueing behavior and the client-visible symptom.
• Inject one 30K-token prompt into a busy interactive load; show the TPOT jitter on other streams, then enable chunked prefill and show the repair.
• Repeat a shared system prefix across requests with prefix caching on and off; quantify the TTFT delta and note how it could flatter a dishonest benchmark.
• Compare the quantized model on the aggregate eval and on a math/code slice; show a slice regression that the average hides.

What to present

One saturation chart (TTFT/TPOT percentiles vs load), the KV arithmetic for your model done by hand, a cost table at three utilization assumptions, one failure trace with its fix, and a half-page build-vs-buy recommendation with explicit reversal conditions. State clearly that all figures are from your own small-scale study — that honesty is itself a senior signal.

Chapter review

Inference is a memory system wearing a model's clothes. Prefill spends compute and sets TTFT; decode spends memory bandwidth and sets TPOT; the KV cache converts context length and concurrency into bytes that compete with weights for HBM. Continuous batching and PagedAttention recover wasted capacity, the quantization ladder trades verified quality for bytes, speculation trades spare compute for latency, and every choice terminates in one number — cost per million tokens at SLO-compliant load — which decides build-versus-buy across Bedrock, Vertex AI, and self-hosted GPU fleets.

Glossary

Prefill

Parallel processing of all prompt tokens that fills the KV cache; compute-bound; determines TTFT.

Decode

One-token-per-pass generation that re-reads weights and KV each step; bandwidth-bound; determines TPOT.

KV cache

Per-layer keys/values stored per token: 2 × layers × KV heads × head_dim × dtype bytes per token.

TTFT / TPOT

Time to first token; time per output token thereafter. Report as percentiles at stated load.

GQA / MLA

Attention variants that shrink KV per token — grouped KV heads, or a cached low-rank latent.

Continuous batching

Iteration-level scheduling: sequences join and leave the batch at every decode step.

PagedAttention

Block-based KV allocation with page tables; kills fragmentation, enables prefix sharing.

Chunked prefill

Splitting long prefills into slices interleaved with decode to bound interference.

Speculative decoding

Draft-then-verify generation that preserves the target distribution exactly via rejection sampling.

RoPE scaling

Rescaling rotary position frequencies (PI, NTK, YaRN) to extend context beyond training length.

MoE

Sparse expert routing: memory priced at total parameters, compute at active parameters.

Goodput

Sustained tokens/s delivered while meeting TTFT/TPOT SLOs — the honest capacity number.

Mastery checklist

• I can derive KV bytes per token for a named model and turn it into a concurrency bound.
• I can explain arithmetic intensity and estimate a batch-1 decode ceiling from bandwidth and model size.
• I can describe what PagedAttention fixed, with the before/after memory-waste numbers.
• I can pick a quantization rung from the bottleneck regime and defend the eval gate around it.
• I can state both preconditions for speculative decoding to pay off — and its exactness guarantee.
• I can explain MoE serving economics: total vs active parameters, and the batch-size erosion effect.
• I can choose a serving engine conditionally and design a fair cross-engine benchmark.
• I can build a cost-per-million-tokens model where utilization is the pivotal variable.
• I can name the managed / provisioned / self-hosted ladder on both AWS and GCP with concrete services.
• I can benchmark TTFT, TPOT, and goodput with load-swept percentiles and name five benchmark deceptions.