Retrieval, Vector Search & Production RAG

1. Start with the retrieval contract

A production RAG system is not “an LLM plus a vector database.” It is an evidence-selection system followed by a constrained answer generator. Define the retrieval contract before choosing an embedding model: given a query, authorization context, freshness boundary, and latency budget, return a ranked set of evidence units with stable identifiers and provenance. Generation may then answer only from those units—or abstain.

flowchart LR
  subgraph SG1["Ingest path"]
    S1["Source of truth"] --> P1["Parse"]
    P1 --> C1["Segment + enrich"]
    C1 --> E1["Embed + index"]
    E1 --> V1["Versioned index"]
  end
  subgraph SG2["Serve path"]
    Q1["Query"] --> A1["Authorize"]
    A1 --> R1["Retrieve candidates"]
    R1 --> F1["Fuse + rerank"]
    F1 --> K1["Pack context"]
    K1 --> G1["Generate + cite"]
  end
  V1 --> R1
  G1 --> OBS["Judgments + traces"]
  R1 --> OBS
  OBS -->|"failure analysis"| C1
    

This decomposition creates useful fault boundaries. If the correct passage never enters the candidate set, prompt tuning cannot recover it. If a relevant passage is retrieved but buried, inspect fusion or reranking. If strong evidence is packed but the answer contradicts it, inspect generation and grounding controls (Chapter 6 covers the evaluation machinery). If the answer is faithful but stale, inspect indexing freshness. If a cross-tenant passage appears, treat it as a security incident, not a relevance defect.

Define the evidence unit

A “document” is rarely the right ranking unit. A policy page may contain a definition, exceptions, a table, and an effective date. Store chunks with document ID, section path, source URI, version, access-control attributes, offsets, content hash, and timestamps. Stable IDs enable deduplication, citation repair, incremental updates, and evaluation across re-indexes. Keep the primary content store authoritative; a search index is usually a derived projection.

2. Design chunks, queries, and context together

Chunk size is not a universal token constant. It mediates two competing risks: small chunks lose the conditions that make a statement true; large chunks dilute the matching signal and consume the generation budget. Start from semantic boundaries—headings, paragraphs, table rows, code symbols, ticket threads—then measure. Store parent relationships so retrieval can find a small unit and context expansion can include the surrounding section.

Query transformation is a hypothesis

Rewriting can normalize spelling, resolve a conversational reference, expand an acronym, or convert a question into search-oriented language; decomposition can retrieve evidence for separate subquestions. Each transformation can also erase a product code, invent intent, or leak unauthorized conversation context. Preserve the original query and protected literals (IDs, dates, jurisdictions, negations), log and version transformations, cap fan-out, and evaluate transformed and untransformed variants on the same judgments. A rewrite that produces fluent language while dropping “EU” or “2026” is a regression, not an improvement.

Pack context as a budgeted ranking problem

Do not concatenate the first k chunks blindly. Deduplicate near-identical passages; group adjacent units; favor coverage of distinct subquestions; reserve tokens for instructions and response; and include provenance outside the quoted text. A simple packing heuristic can maximize reranker score plus subtopic coverage minus redundancy and token cost. Evaluate whether evidence survives packing, not merely whether retrieval found it—packing is also the dominant lever on per-answer generation cost (Section 7).

3. Contextual retrieval and late chunking

Classic chunking has a structural defect: a chunk is embedded in isolation, so pronouns, abbreviations, and section-scoped conditions lose their referents. “The termination clause above does not apply to contractors” embeds poorly when “above” is in a different chunk. Two 2024–2025 techniques attack this at ingest time and are now standard interview material.

Contextual retrieval, described by Anthropic, uses an LLM at ingestion to generate a short document-situating preamble for each chunk (“This clause is from the 2026 EU contractor policy, section 4, on notice periods…”), prepends it before embedding, and does the same for the BM25 index. Anthropic reports roughly a 49% reduction in top-20 retrieval failure rate for contextual embeddings plus contextual BM25, and about 67% when a reranker is added—vendor-reported numbers, but directionally consistent with public replications. The cost is one LLM call per chunk at ingest; prompt caching (Chapter 2) makes this cheap because the full document is the cached prefix and only the chunk varies.

Late chunking, introduced by Jina AI, inverts the order of operations: run the whole document through a long-context embedding model once, then mean-pool token embeddings per chunk boundary afterward. Each chunk vector is conditioned on the surrounding text without any extra LLM calls—but it requires an embedding model with a long context window and does not improve the lexical index.

flowchart TD
  D["Full document"] --> N1["Naive: split, then embed each chunk alone"]
  D --> C2["Contextual: prepend LLM-generated document context, then embed"]
  D --> L3["Late chunking: embed whole document, then pool token vectors per chunk"]
  N1 --> X1["Chunk vector loses global references"]
  C2 --> X2["Chunk carries entity, section, and date context"]
  L3 --> X3["Chunk conditioned on surrounding text at no LLM cost"]
    

4. Dense, sparse, hybrid, and reranked retrieval

Sparse lexical retrieval rewards token overlap and is especially strong for identifiers, names, rare terms, and exact phrases. BM25-style scoring balances term frequency, document frequency, and length normalization. Dense retrieval maps queries and passages into a vector space, recovering conceptual similarity and paraphrase. Neither dominates across all query types. Hybrid retrieval builds independent candidate lists and combines them.

Fuse ranks before comparing incompatible scores

Dense cosine scores and sparse scores do not share a calibrated scale. A raw weighted sum can be dominated by whichever retriever emits larger values. Reciprocal rank fusion (RRF) uses positions instead: for each document, add 1 / (c + rank) across result lists, where c dampens top-rank differences. Qdrant’s Query API supports hybrid and multi-stage retrieval with prefetches and RRF/DBSF fusion; Elasticsearch likewise documents RRF as a hybrid-search fusion option. Tune weighted fusion on held-out judgments, not intuition.

# Illustrative, one-based ranks and c = 60
dense = {"policy-A": 1, "faq-B": 2, "policy-C": 3}
sparse = {"policy-C": 1, "policy-A": 2, "memo-D": 3}

def rrf_score(doc_id):
    ranks = [ranking[doc_id] for ranking in (dense, sparse)
             if doc_id in ranking]
    return sum(1 / (60 + rank) for rank in ranks)

# policy-A and policy-C gain support from both lists.

RRF is robust but discards score magnitude. Score-distribution normalization or learned fusion can exploit more information, but it needs validation and drift monitoring. Always retrieve deeper than the final k; fusion and reranking cannot select a document absent from every candidate list.

Rerank selectively

A cross-encoder or LLM reranker jointly inspects query and candidate and can resolve nuanced constraints. It adds cost and tail latency, and a reranker cannot repair missing candidates. Cache only when the query, corpus/index version, authorization scope, and reranker version make reuse safe. Test a cheap deterministic reranker or metadata boost as a baseline before adding another model call.

5. Vector geometry and approximate nearest neighbors

The similarity function must match model training and stored-vector treatment. Cosine compares direction; dot product combines direction and magnitude; Euclidean distance measures geometric separation. For unit-normalized vectors, ranking by cosine and dot product is equivalent, and squared Euclidean distance is monotonically related. Do not normalize reflexively if magnitude carries trained meaning. Record model, dimensions, preprocessing, normalization, and distance metric as one versioned contract.

Exact search is the quality oracle

An exact scan computes distances against all eligible vectors and provides the reference neighbor set for measuring approximate recall. It is often practical for small or tightly filtered subsets. Approximate nearest-neighbor (ANN) indexes trade perfect recall for lower latency and resource use. Keep an exact path in the benchmark environment; without it, you cannot tell whether missed results come from the embedding or the index.

flowchart TD
  subgraph SG3["Layer 2 - sparsest"]
    EPT["Entry point"] --> H1["Greedy hop toward query"]
  end
  subgraph SG4["Layer 1 - denser"]
    H2["Descend and refine locally"]
  end
  subgraph SG5["Layer 0 - full graph"]
    H3["Explore ef_search candidates"] --> RES["Top-k approximate neighbors"]
  end
  H1 --> H2
  H2 --> H3
    

Hierarchical Navigable Small World (Malkov & Yashunin) constructs a multi-layer proximity graph. m controls graph connectivity and therefore memory/build/search behavior; ef_construct expands the build-time candidate pool; ef_search (often surfaced as hnsw_ef or ef) expands the query-time search. Higher values commonly improve recall but cost build time, memory, or latency. Benchmark the actual filtered workload rather than repeating defaults. Google’s ScaNN takes a different route—partitioning plus anisotropic quantization—and underlies Vertex AI Vector Search and AlloyDB’s ANN index (Section 10); the tuning story is different, the recall-versus-latency discipline identical.

Filtering changes the graph problem

A post-filter may leave too few candidates; a strict filter can also make graph traversal ineffective. Build indexes for frequent metadata filters and test selectivity slices. Qdrant documents a filterable HNSW approach and recommends creating payload indexes before ingestion so filter-aware graph edges can be built. In pgvector, approximate-index filtering is applied after the scan; its README describes iterative scans as a way to search farther when filtering removes candidates. These implementation differences belong in a database decision and benchmark.

Quantization is an end-to-end trade

Quantization compresses vector representations to reduce memory and often accelerate distance work, at the cost of approximation error. Preserve original vectors when a two-stage search can rescore a larger compressed candidate set. Qdrant documents scalar, product, and binary approaches and explicitly frames the choice as accuracy, storage, and speed. Measure relevance, ANN recall against exact search, p50/p95/p99 latency, build time, and resident memory—not only compression ratio. The arithmetic is a first-class cost lever: see Section 7.

6. Beyond flat retrieval: GraphRAG, multimodal, and agentic loops

Flat top-k retrieval assumes the answer lives in a handful of independently rankable passages. Three query families break that assumption, and by 2026 interviewers expect you to know which upgrade fixes which family—and what each one costs.

GraphRAG: when relationships are the evidence

Microsoft’s GraphRAG uses an LLM at ingest to extract entities and relations into a knowledge graph, clusters it into communities, and pre-summarizes each community. “Local” queries traverse from an entity through its neighborhood; “global” queries (“what are the recurring risk themes across these filings?”) map over community summaries—questions flat retrieval simply cannot answer because no single passage contains the answer. The price is steep: LLM extraction over the whole corpus at ingest, graph maintenance on every update, and a much harder evaluation problem. The honest default remains flat hybrid retrieval; reach for graphs when queries are genuinely multi-hop or aggregative and the corpus is entity-dense (compliance, biomedical, org knowledge, incident histories).

flowchart LR
  DOCS["Corpus"] --> EX["LLM entity + relation extraction"]
  EX --> KG["Knowledge graph"]
  KG --> CM["Community detection + summaries"]
  QL["Local query about one entity"] --> KG
  QG["Global query about themes"] --> CM
  KG --> AN1["Neighborhood evidence"]
  CM --> AN2["Corpus-level synthesis"]
    

Multimodal RAG: tables, figures, and ColPali

Enterprise answers hide in tables, charts, and scanned diagrams that text parsers mangle. Two viable strategies: (a) parse-and-describe—extract tables as structured units, generate text summaries of figures, embed both alongside the source crop; (b) skip parsing entirely with vision retrievers like ColPali, which embeds page screenshots as grids of patch vectors via a vision-language model and scores queries with late interaction (MaxSim over multivectors). ColPali-style retrieval is remarkably robust on visually rich PDFs and eliminates the parser as a failure mode—but multivector storage is an order of magnitude larger per page, and your engine must support multivector comparators natively (Qdrant does; see Section 9). Retrieval returns page images, so the generator must be a vision-capable model, which raises per-answer token cost.

Agentic retrieval: iterate only when single-shot fails

Single-shot retrieval fails on ambiguous, multi-hop, or under-specified questions where first-pass recall is inherently low. Agentic (iterative) retrieval—in the spirit of Self-RAG and FLARE—lets the model assess evidence, reformulate, pivot to another index or tool, and stop when coverage is sufficient. It typically multiplies latency and token cost by 2–5× and compounds error if the assessment step is weak, so bound it: a fixed step budget, deterministic stop conditions, and abstention on budget exhaustion. Chapter 5 covers the surrounding agent machinery; here the retrieval-side rule is that every iteration must run through the same authorization and evaluation contract as the first.

flowchart LR
  U["User question"] --> PL["Plan retrieval step"]
  PL --> RT["Retrieve via contract"]
  RT --> AS["Assess evidence coverage"]
  AS -->|"sufficient"| ANS["Answer with citations"]
  AS -->|"gap found"| RF["Reformulate or pivot source"]
  RF -->|"budget left"| PL
  RF -->|"budget exhausted"| AB["Abstain or partial answer"]
    

7. Freshness, semantic caching, and RAG cost engineering

Three production concerns dominate senior RAG interviews in 2026 and rarely appear in tutorials: keeping the index true to a moving corpus, not paying for the same answer twice, and knowing where the money actually goes.

Freshness is an SLO, not a batch job

Treat “time from source change to retrievable” as a first-class SLO. The reference pattern: change data capture on the source of truth → queue → re-parse and re-chunk only changed documents (content hashes decide) → upsert by deterministic ID → tombstone deletes → verify visibility. Deletions are the part teams forget: a revoked document that still answers queries is a compliance incident. Track freshness lag as a monitored metric with alerting, and record the embedding model version alongside content version so a model migration and a content update cannot be confused.

• Freshness lag — measure per-document time from source commit to index visibility; alert on the p95, not the mean.
• Idempotent upserts — deterministic point IDs plus content hashes make reprocessing safe and reconciliation possible.
• Deletes and ACL changes — propagate with higher priority than inserts; stale permissions are security bugs.
• Re-embed selectively — hash-gated incremental embedding avoids full-corpus re-embeds on every pipeline run.

Semantic caching of answers

Support and internal-helpdesk traffic is heavily repetitive; a semantic cache embeds incoming queries and serves a stored answer when a previous query is similar enough. Done naively it is a correctness and security hazard: paraphrases with different constraints (“2025” vs “2026”, negations, tenants) collide, and corpus updates silently invalidate cached answers. The guards are the design: scope cache keys by tenant, corpus version, and policy version; require a high similarity threshold plus a cheap lexical constraint check on protected literals; TTL tied to the freshness SLO; invalidate entries whose cited sources changed; and cache only high-confidence answers. Hit rates of 20–40% on repetitive support workloads are a realistic example planning number—measure your own. (This is distinct from provider-side prompt/prefix caching, covered in Chapter 2, and from reranker result caching in Section 4.)

flowchart LR
  Q["Incoming query"] --> EMB["Embed query"]
  EMB --> LK["Similarity lookup in answer cache"]
  LK -->|"hit above threshold"| GD["Guards: tenant, corpus version, literals, TTL"]
  GD -->|"pass"| CA["Serve cached answer"]
  GD -->|"fail"| FULL["Full retrieve + generate"]
  LK -->|"miss"| FULL
  FULL --> WR["Write answer + query vector back"]
  WR --> LK
    

Where the money goes

Run the arithmetic before optimizing. Example only: 10M chunks at 1024 dimensions in float32 is 10M × 1024 × 4 B ≈ 41 GB of raw vectors—int8 scalar quantization cuts it to ~10 GB, binary to ~1.3 GB with rescoring, which decides whether the index fits RAM or needs disk-backed storage. Embedding 10M chunks of ~300 tokens at an example $0.02/M tokens is roughly $60 one-time—ingest embedding is rarely the problem. Generation dominates steady-state: a 6k-token packed prompt at an example $3/M input is about $0.018 per answer, hundreds of times the marginal vector-query cost on a warm node. So the levers, in order of typical impact: cache answers, pack fewer/better tokens (reranking pays for itself here), route easy queries to cheaper models (Chapter 2), then quantize and tier storage.

8. Benchmark relevance and performance together

A golden set contains representative queries plus graded or binary relevance judgments over evidence units. Sample navigational, exact-identifier, semantic, multi-hop, filtered, multilingual, fresh-content, long-document, and “no answer” cases. Split tuning from final validation so fusion weights and chunk sizes are not optimized on the score you report. Version queries, judgments, corpus snapshot, parsing, embedding, index configuration, and code.

Report macro averages and slices. A 2-point nDCG gain that hides a severe regression on one tenant, language, or exact-ID query is not a safe improvement. Inspect per-query deltas and categorize failures: parse loss, chunk boundary, stale index, ACL/filter error, candidate miss, fusion error, reranker error, packing loss, or generator misuse. New variants from this chapter—contextual retrieval, GraphRAG, agentic loops, the semantic cache—enter the harness as configurations, never as unconditioned defaults.

Measure under concurrency

Benchmark isolated stage latency and end-to-end latency. Warm and cold behavior differ. Include index build and freshness lag, throughput, error rate, CPU, memory, I/O, and cost per successful answer. Run controlled sweeps—candidate depth, ef, quantization, reranker depth—with fixed corpus and query set. Then load-test the best few configurations because tail latency can change under resource contention.

9. Qdrant as a production case study

Qdrant’s core data model is a collection of points, where a point has an ID, one or more vectors, and optional JSON payload. Named vectors allow different representations on the same point, and multivector fields with a MaxSim comparator support ColPali-style late-interaction retrieval natively—one reason it pairs well with the multimodal patterns in Section 6. Distance and dimensions are configured per vector. Payload fields support filtering; index frequent, security-relevant fields such as tenant or visibility before ingestion. The collection documentation notes that point and indexed-vector counters can be approximate during optimization, so do not use them as an exact ingestion ledger.

Collection and tenancy choices

A collection per tenant gives strong operational separation but can create excessive collection/index overhead and complicate fleet-wide updates. A shared collection with tenant payload and a mandatory filter is efficient for many smaller tenants but makes authorization enforcement and noisy-neighbor testing critical. Dedicated collections may be appropriate for very large tenants, incompatible schemas, independent scaling, or embedding migrations. Put authorization-derived filters in trusted server code, not model output or user-provided query text.

Storage, index, and availability

Choose in-memory versus on-disk vectors and HNSW, quantization, and rescoring from the measured working set—the Section 7 memory arithmetic decides which regime you are in. Qdrant’s optimization guide documents configurations for low memory, high speed, and high precision; these are starting scenarios, not automatic recommendations. Sharding increases capacity and parallelism; replication improves availability and can increase read capacity, but multiplies storage and write work. The distributed-deployment documentation notes that self-hosted shard balancing is operational work and recommends a load balancer so replicas and coordinators are not stranded behind one entry node. On Kubernetes, treat it as a stateful system: persistent volumes, anti-affinity or topology spread, resource limits, disruption budgets, snapshot/restore drills, and an upgrade/rollback runbook—stateful recovery and shard placement, not a green Pod, define readiness.

Writes, backups, and migrations

Use deterministic point IDs and content hashes so reprocessing is idempotent. Write a source version into payload. Reconcile source-of-truth counts and hashes, not approximate index counters. Exercise snapshot creation and restore; a backup that has never restored is only an assumption. For an embedding change, build a new named vector or collection, backfill from the authoritative content, dual-read a shadow sample, compare quality and latency, switch an alias or routing layer, and retain rollback until freshness and parity checks pass. Qdrant provides official guides for snapshots and zero-downtime embedding-model migration; validate version-specific mechanics before execution.

10. Managed retrieval on AWS and GCP — and when to build instead

Both clouds now sell the entire Figure 1 pipeline as a service. A senior candidate is expected to know what each managed layer actually does, where its control surface ends, and how to argue build-vs-managed without ideology. Chapters 8 and 9 cover the full stacks; here is the retrieval slice.

Build vs managed, honestly

The deciding questions are always the same: Is retrieval quality your product differentiator or a commodity? Can the managed chunking/ranking be overridden where your judgments show it failing? What does exit cost look like—can you re-derive the index from the authoritative store you kept? Answer those with benchmark evidence on your corpus, and the choice usually makes itself.

11. Select, migrate, and debug the whole system

Choose a search engine by workload, not category labels. A dedicated vector system is attractive for vector-native filtering, multivector search, and independent scaling. Elasticsearch/OpenSearch can consolidate mature lexical search, aggregations, and hybrid retrieval. PostgreSQL with pgvector can minimize operational surface when transactional metadata and scale fit one system. Apache Solr still anchors many Lucene-based estates; a Solr migration must translate analyzers, synonyms, boosts, faceting, and operational SLAs—relevance behavior and recovery procedures, not just stored documents. Include team expertise, recovery, tenancy, write patterns, compliance, cost, and migration reversibility in every comparison.

A safe migration sequence

1. FreezeVersion the retrieval contract, judgments, and a replayable traffic sample.
2. BackfillLoad the target from the authoritative source with deterministic IDs; reconcile counts and hashes.
3. ShadowCompare result overlap, relevance, filters, latency, and errors without affecting users.
4. Dual-writeOr capture a change log; monitor freshness divergence between old and new.
5. CanaryRoute by tenant/query class with automatic rollback gates on relevance and p95 deltas.
6. Cut overVerify restore/DR and dashboards; retire the old index only after the rollback window.

Search quality monitoring in production needs proxies plus sampled judgments. Track empty/low-confidence results, reformulations, citation clicks, answer abstentions, retrieval overlap by version, semantic-cache hit/invalidation rates, freshness lag, and user feedback. Do not equate click-through with relevance: position, presentation, and user urgency confound it. Convert investigated failures into the offline dataset.

Interview playbook

For a retrieval design question, use RANKED:

1. R — Requirements: users, corpus, relevance definition, freshness SLO, ACLs, scale, latency, and cost per answer.
2. A — Authoritative data: source, parsing, stable IDs, versions, lineage, and deletion behavior.
3. N — Nomination: lexical/dense candidate generators, contextual enrichment, filters, depths, and exact baseline.
4. K — Keep order: fusion, reranking, deduplication, parent expansion, and packing.
5. E — Evaluate and expose: judgments, slices, metrics, traces, load tests, and error taxonomy.
6. D — Deploy safely: idempotent writes, sharding/replication, caching guards, backups, canary, rollback, and SLOs.

Then earn senior credit by escalating deliberately: name the flat-RAG baseline first, and justify each upgrade—contextual retrieval, GraphRAG, multimodal, agentic loops, semantic caching—by the query slice it fixes and the cost it adds. On cloud questions, show you know what Bedrock Knowledge Bases or Vertex AI Search actually manage, and where their control surface ends.

Common traps

• Calling cosine similarity “accuracy,” or conflating ANN recall with relevance recall.
• Tuning on a few memorable queries and reporting the same queries as validation.
• Combining raw dense and sparse scores without calibration, instead of rank fusion.
• Applying tenant filters after retrieval, allowing unauthorized candidates into context or traces.
• Assuming a larger chunk, larger k, or larger context window monotonically improves answers.
• Proposing GraphRAG or agentic loops before showing that flat hybrid retrieval fails a measured slice.
• Adding a semantic cache with no tenant/version scoping—an availability feature that becomes a correctness or security bug.
• Choosing managed vs self-hosted by ideology rather than control-surface, cost, and exit analysis.
• Describing a migration as “reindex and switch” with no change capture, parity check, canary, or rollback.

Question bank

These prompts test retrieval reasoning, not memorized product vocabulary.

Proof artifact: a reproducible hybrid-search benchmark

Build a small, public-data search service that makes retrieval quality, filtered performance, and operational behavior inspectable. All results are portfolio measurements, not claims about Purnendu’s production experience.

Steps

1. Select a legally usable corpus with meaningful structure. Freeze a corpus manifest containing source URI, content hash, version, and parser result.
2. Create 150–300 queries with binary or graded evidence judgments. Include exact IDs, paraphrases, strict metadata filters, multiple required passages, stale versions, and no-answer cases. Split tuning and validation.
3. Implement five fixed variants: sparse baseline; dense exact/ANN; hybrid RRF; hybrid plus reranking; hybrid plus contextual retrieval. Keep parsing and corpus constant.
4. Sweep chunking, candidate depth, HNSW search effort, filter selectivity, and optional quantization. Record every configuration.
5. Add a guarded semantic answer cache and replay a repetitive traffic sample; report hit rate, false-hit rate on judged paraphrases, and cost saved per answer.
6. Run single-request and concurrent load profiles on declared hardware. Capture stage spans, p50/p95/p99, throughput, errors, CPU, memory, index size, and freshness lag.
7. Run the same golden set through one managed path (Bedrock Knowledge Bases or Vertex AI Search) and write a decision memo comparing it and Qdrant against an adjacent option such as pgvector, including a reversal condition.

Metrics

Report precision@5, recall@20, MRR, nDCG@10, evidence coverage, no-answer behavior, ANN recall against exact search, filtered-query slices, latency percentiles, index/build time, memory, cache hit/false-hit rates, and cost per successful answer. Show per-query deltas, confidence intervals or paired resampling when practical, and the error taxonomy—not only averages.

Deliberate failures

• Remove the payload index for a frequent strict filter and observe quality/latency under load.
• Lower ANN search effort until exact-neighbor recall visibly fails, then distinguish ANN loss from embedding relevance.
• Corrupt a parser boundary so an exception is separated from a policy statement; confirm the golden set catches it.
• Pause incremental indexing while source versions advance; ensure freshness monitoring and a user-visible policy respond.
• Update a document cited by cached answers without invalidating the semantic cache; show the stale-answer detection catching it.
• Make the reranker unavailable; verify timeout, fallback ranking, trace status, and bounded latency.

What to present

Present one pipeline diagram, the dataset card, a Pareto chart of nDCG versus p95 latency, the cost-per-answer decomposition, two failure traces, and the one-page engine decision. Demonstrate one query where sparse wins, one where dense wins, one where contextual retrieval rescues an isolated chunk, one where a filter breaks naïve ANN, and one where the system correctly abstains.

Chapter review

Production retrieval is a chain of contracts. Preserve authoritative content and provenance, generate complementary candidates, fuse and rerank deliberately, pack evidence within a budget, and measure component as well as end-to-end behavior. Escalate beyond flat retrieval—contextual enrichment, graphs, vision retrievers, agentic loops, semantic caches—only when a measured query slice justifies the added cost, and choose between managed platforms and self-hosted engines on control-surface, cost, and exit evidence rather than ideology.

Glossary

ANN recall

Fraction of exact nearest neighbors recovered by an approximate index at a cutoff; distinct from judged relevance recall.

BM25

Lexical ranking family using term frequency, inverse document frequency, and document-length normalization.

Contextual retrieval

Prepending an LLM-generated document-situating preamble to each chunk before embedding and lexical indexing.

Late chunking

Embedding a whole document with a long-context model, then pooling token vectors per chunk boundary afterward.

GraphRAG

Retrieval over an LLM-extracted knowledge graph with community summaries, enabling local entity and global synthesis queries.

Late interaction

Scoring queries against multivector representations (for example MaxSim over ColPali patch vectors) instead of one pooled vector.

HNSW

Hierarchical proximity-graph ANN structure with build, memory, latency, and recall trade-offs.

RRF

Reciprocal rank fusion, which combines result lists using document positions rather than raw score scales.

Reranker

Later-stage scorer that evaluates a query and candidate more jointly than a first-stage retriever.

Semantic cache

Answer cache keyed by query-embedding similarity, guarded by tenant, corpus version, literals, and TTL.

Freshness lag

Time from a source-of-truth change to that change being retrievable; an SLO, not an accident.

Evidence coverage

Whether packed context contains all facts required to answer a task, not merely one relevant chunk.

Mastery checklist

• I can isolate ingestion, candidate, rank, pack, and generation failures.
• I can give a query where sparse wins and one where dense wins.
• I can calculate a simple RRF result and explain when it is insufficient.
• I can explain contextual retrieval and late chunking, including their cost asymmetry and when each pays.
• I can separate embedding relevance, ANN recall, and end-to-end answer quality.
• I can explain how HNSW parameters and strict filters change the workload.
• I can name the query slices that justify GraphRAG, ColPali-style retrieval, and agentic loops—and their costs.
• I can design a guarded semantic cache and a freshness pipeline with deletion handling.
• I can decompose RAG cost per answer and order the levers by impact.
• I can map a retrieval design onto Bedrock/OpenSearch/Kendra and Vertex AI Search/Vector Search/AlloyDB and defend build-vs-managed.
• I can outline a shadowed, canaried, reversible search migration.