AI Engineering
Interview Handbook

1. Preparation is evidence selection

The syllabus is deliberately not a survey of artificial intelligence. It is a conversion plan: turn existing production-GenAI familiarity into evidence for senior, globally remote Applied AI, AI Platform, Forward Deployed, and Architecture interviews. The highest-return work is therefore not memorizing more definitions. It is choosing which claims you want an interviewer to believe and constructing honest, inspectable support for each claim.

Use a three-layer model. A signal is what the role needs to infer—for example, “can tune retrieval under latency constraints.” Evidence is what makes that inference reasonable—a benchmark, design, code review, incident analysis, or accurately described past decision. A delivery is how the evidence appears in the interview: a two-minute story, a whiteboard, a working repository, or a written memo. Weak preparation jumps directly from a topic name to a rehearsed explanation. Strong preparation joins all three layers.

The five-part readiness test

For every priority topic, ask whether you can define it precisely, design it under constraints, implement a small version, debug a deliberate failure, and defend one consequential trade-off. A topic is not interview-ready until all five are plausible. This prevents the common illusion that reading documentation equals being able to reason in a live system-design conversation.

2. Convert each role into a signal matrix

Job titles are noisy; recurring responsibilities are more useful. Extract verbs, objects, constraints, and collaboration surfaces from each current posting. “Design,” “operate,” “migrate,” “evaluate,” and “advise customers” imply different evidence from “experiment with.” Re-check live postings before applying because requirements change. Automattic, for example, describes role-specific assessments and a paid trial in its official hiring process, while Canonical currently lists globally home-based MLOps field and reliability roles; both reward demonstrable execution and communication, not keyword coverage.

Label the evidence honestly

Mark each cell experienced, built for practice, understood but not operated, or gap. Those labels are valuable. A credible statement such as “I have not operated Qdrant at that scale; here is the benchmark and failure analysis I built to learn the operating trade-offs” is stronger than an inflated production claim. Never present a tutorial metric, synthetic benchmark, team result, or hypothetical design as your personal outcome.

3. Manage preparation as a weighted portfolio

The source syllabus proposes a useful default allocation: 20% retrieval and search; 15% each for evaluation, enterprise integration, reliability, and system design/FDE; and 10% each for coding/SQL and leadership/writing. Treat these as starting weights, not a rigid calendar. Multiply each weight by three factors: role frequency, current evidence weakness, and interview proximity.

priority(topic) = role_frequency × evidence_gap × interview_urgency

# Example only: 1 (low) to 3 (high)
retrieval = 3 × 2 × 3 = 18
model-training theory = 1 × 2 × 1 = 2

The calculation is intentionally simple. Its purpose is to expose opportunity cost. If a role repeatedly mentions hybrid search, relevance evaluation, migrations, and customer troubleshooting, another evening on obscure training mathematics is probably a poor trade unless the posting explicitly asks for it. Preserve a small breadth budget so you can discuss adjacent systems, but spend the main budget where proof changes the hiring decision.

A weekly allocation that produces outputs

Plan in deliverables rather than hours. One week might end with: a retrieval benchmark and chart; a design interview recording; two async-Python exercises; one SQL query review; and one rewritten leadership story. These are completion criteria. “Study vector databases for six hours” is not. Reserve roughly one practice block in five for mixed, timed rehearsal because interviews require retrieval of knowledge under uncertainty, not only careful construction at leisure.

4. Build a proof stack and a story ledger

A proof stack is a small set of artifacts that covers many signals without becoming a portfolio museum. The most efficient centerpiece for this syllabus is a production-shaped RAG service: ingestion, dense and sparse retrieval, filters, reranking, citations, an evaluation harness, traces, deployment controls, and a design memo. Each layer supports a different interview, while the shared scenario lets you reason deeply instead of maintaining unrelated demos.

Artifacts at four levels

1. Code proof: readable implementation, tests, typed boundaries, reproducible commands, and dependency versions.
2. Decision proof: an architecture diagram and short ADR stating constraints, alternatives, choice, and reversal conditions.
3. Measurement proof: dataset provenance, metric definitions, baseline, variant, uncertainty, latency, and cost.
4. Operations proof: traces, SLO proposal, capacity assumptions, injected failure, alert, rollback, and runbook.

Keep a separate story ledger for real experience. For every candidate story, record: situation and stakes; exact personal responsibility; collaborators; decision alternatives; action; observable result; what you would change; and which claims need qualification. Replace confidential names and values only when necessary, and state that values are rounded or anonymized. If you lack a metric, say what was observed and what you would measure now—do not reverse-engineer an impressive number.

5. Answer at senior scope

Senior answers reveal a decision process. Begin by locating the goal, users, risk, and constraints. Establish a simple baseline. Decompose the system and make boundaries explicit. Compare alternatives against criteria. Cover failure, security, rollout, observability, and ownership. Close with validation and what would change your mind. This sequence works in system design, troubleshooting, project deep dives, and customer discovery.

Separate facts, assumptions, and decisions

Say “the requirement states,” “I am assuming,” and “I would choose” rather than blending all three. Ask a small number of high-value clarifying questions, then proceed with declared assumptions. In a 45-minute design session, exhaustive discovery is impossible; the signal is whether the assumptions are consequential and whether the design can adapt.

Make influence observable

Staff-level influence is not a title or a meeting count. Explain the mechanism: you wrote an options memo, instrumented a disputed bottleneck, facilitated a review, created an adoption path, mentored an owner, or changed a standard. Explain the resistance and how feedback altered the plan. “I convinced everyone” is weaker than a transparent decision process that lets reasonable people converge.

6. Run a practice-and-feedback operating system

Preparation needs short feedback loops. Use a weekly cycle: diagnose, build, rehearse, review, and update. Every mock should produce one or two observable corrections—“state assumptions before drawing,” “define the denominator of recall,” or “name retry ownership”—not a vague instruction to be more senior.

Use a stop-doing list

Stop adding notes once a topic can pass the five-part readiness test. Stop polishing visual design when the artifact lacks a baseline. Stop collecting model trivia that changes weekly unless the role requires vendor depth. Stop rehearsing only successful stories. Failure analysis is where operational judgment becomes visible.

Syllabus checkpoint: make every outcome measurable

The preparation PDF repeatedly asks for evidence across quality, latency, cost, uptime, and customer impact. Treat these as a balanced evidence set rather than choosing whichever number looks best. Quality needs a defined task and dataset; latency needs a percentile, load, and boundary; cost needs a unit such as cost per successful task; uptime needs an SLI and time window; customer impact needs an observable workflow or business outcome.

If exact production values are confidential, use approved ranges or explain the measurement method and direction. Never invent precision. The senior signal is the causal chain from decision to measurable outcome—and an honest account of confounders.

Interview playbook

Use the CONTEXT answer frame when a question is broad:

1. C — Customer and consequence: Who needs what, and what happens if the system is wrong or late?
2. O — Objectives and constraints: Define quality, latency, cost, privacy, scale, and timeline.
3. N — Narrow baseline: Choose the smallest system that can test the value proposition.
4. T — Trade-offs and topology: Draw boundaries, compare alternatives, and justify the choice.
5. E — Exceptions: Cover dependency failure, bad data, abuse, and partial completion.
6. X — eXecution: Explain sequencing, ownership, migration, approvals, and communication.
7. T — Tests and telemetry: State acceptance gates, SLOs, feedback, and reversal conditions.

Common traps

• Technology-first: naming a vector database or agent framework before defining the problem.
• Unbounded “we”: making team output sound like personal implementation. State your role explicitly.
• Metric theater: quoting a percentage without dataset, denominator, baseline, or business meaning.
• Perfect hindsight: omitting uncertainty, disagreement, or what changed during delivery.
• Demo scope: ignoring authorization, tenant isolation, retries, evaluation, rollout, or on-call ownership.
• Vendor recital: describing features instead of criteria, measurement, and failure behavior.

For experience questions, use Situation → Stakes → My responsibility → Options → Action → Result → Reflection. If a result is qualitative, keep it qualitative. If discussing a portfolio exercise, label it as such before presenting any numbers.

Question bank

Practise aloud. Each answer should use evidence appropriate to its claim and should survive the probes without invented detail.

Proof artifact: the interview evidence dossier

Create a version-controlled dossier that maps one target role to inspectable proof. This artifact tests preparation discipline; it is not a claim about Purnendu’s past outcomes.

Build steps

1. Save a dated copy or structured summary of one live job description. Extract no more than ten decision-relevant signals.
2. Create a matrix with columns for signal, importance, evidence label, artifact/story, limitation, and next action.
3. Add one architecture diagram, one benchmark report, one evaluation report, one failure/runbook note, and one one-page decision memo. Link each to the signals it supports.
4. Create six real-experience story cards. Record exact personal scope and mark every number as exact, rounded/anonymized, or unavailable.
5. Record a 45-minute design mock and two-minute project introduction. Score both with the same rubric, then repeat one week later.
6. Publish a short README that explains reproducibility, secrets/privacy boundaries, and what the practice system does not prove.

Metrics to capture

• Coverage: percentage of top signals with at least one inspectable proof; report gaps separately rather than hiding them in an average.
• Readiness: count of topics passing all five tests—define, design, implement, debug, defend.
• Delivery: rubric scores for framing, assumptions, trade-offs, failure handling, measurement, and clarity.
• Claim hygiene: number of unqualified “we” statements, unsupported metrics, or practice results phrased as experience. Target zero.
• Cadence: planned versus completed proof outputs and repeated prompts, not passive study hours.

Deliberate failure injection

Remove the baseline and dataset description from the benchmark report, then ask a reviewer to interpret the claimed gain. The result should become non-actionable. Restore the missing context and record the difference. Next, rehearse one story with deliberately ambiguous ownership; use the recording to locate every sentence where “we” conceals personal responsibility. These failures demonstrate why measurement provenance and claim boundaries are part of engineering quality.

What to present

Present the one-page signal matrix first, then one proof chain end to end: role need → decision → artifact → metric → failure → lesson. Use example numbers only if clearly labeled as hypothetical or portfolio measurements. Keep the complete repository available, but do not force an interviewer through every file.

Chapter review

Interview preparation is a constrained engineering program. Start with live role signals, allocate effort by evidence gap, produce a compact proof stack, rehearse senior decision-making, and use measured feedback. The goal is not to sound universally expert. It is to make relevant, honest competence easy to verify.

Glossary

Hiring signal

The conclusion an interviewer needs to draw about capability, judgment, or collaboration.

Proof artifact

An inspectable output—code, test, benchmark, diagram, memo, or runbook—that supports a specific signal.

Evidence label

An explicit distinction among production experience, practice implementation, conceptual understanding, and a gap.

Reversal condition

New evidence or a threshold that would cause a technical decision to be revisited.

Story ledger

A claim-calibrated inventory of real experiences, responsibilities, results, and lessons.

Counter-metric

A measure that exposes harmful optimization of a primary metric, such as latency beside retrieval quality.

Mastery checklist

• I can derive a ranked backlog from a current job description.
• I can label each claim as experience, portfolio proof, conceptual knowledge, or gap.
• I have one artifact each for code, decisions, measurement, and operations.
• I can explain one system through goal, constraints, alternatives, failure, rollout, and measurement.
• I have six stories with exact personal scope and no invented metrics.
• I can deliver two-minute and six-minute versions of my strongest project story.
• I have repeated a mock prompt after applying rubric-based feedback.
• I can name what I deliberately deferred and the trigger for revisiting it.

1. Start with the retrieval contract

A production RAG system is not “an LLM plus a vector database.” It is an evidence-selection system followed by a constrained answer generator. Define the retrieval contract before choosing an embedding model: given a query, authorization context, freshness boundary, and latency budget, return a ranked set of evidence units with stable identifiers and provenance. Generation may then answer only from those units—or abstain.

This decomposition creates useful fault boundaries. If the correct passage never enters the candidate set, prompt tuning cannot recover it. If a relevant passage is retrieved but buried, inspect fusion or reranking. If strong evidence is packed but the answer contradicts it, inspect generation and grounding controls. If the answer is faithful but stale, inspect indexing freshness. If a cross-tenant passage appears, treat it as a security incident, not a relevance defect.

Define the evidence unit

A “document” is rarely the right ranking unit. A policy page may contain a definition, exceptions, a table, and an effective date. Store chunks with document ID, section path, source URI, version, access-control attributes, offsets, content hash, and timestamps. Stable IDs enable deduplication, citation repair, incremental updates, and evaluation across re-indexes. Keep the primary content store authoritative; a search index is usually a derived projection.

2. Design chunks, queries, and context together

Chunk size is not a universal token constant. It mediates two competing risks: small chunks lose the conditions that make a statement true; large chunks dilute the matching signal and consume the generation budget. Start from semantic boundaries—headings, paragraphs, table rows, code symbols, ticket threads—then measure. Store parent relationships so retrieval can find a small unit and context expansion can include the surrounding section.

Query transformation is a hypothesis

Rewriting can normalize spelling, resolve a conversational reference, expand an acronym, or convert a question into search-oriented language. Decomposition can retrieve evidence for separate subquestions. Each transformation can also erase a product code, invent intent, or leak unauthorized conversation context. Preserve the original query, log transformations, cap fan-out, and evaluate transformed and untransformed variants on the same judgments.

Pack context as a budgeted ranking problem

Do not concatenate the first k chunks blindly. Deduplicate near-identical passages; group adjacent units; favor coverage of distinct subquestions; reserve tokens for instructions and response; and include provenance outside the quoted text. A simple packing heuristic can maximize reranker score plus subtopic coverage minus redundancy and token cost. Evaluate whether evidence survives packing, not merely whether retrieval found it.

3. Dense, sparse, hybrid, and reranked retrieval

Sparse lexical retrieval rewards token overlap and is especially strong for identifiers, names, rare terms, and exact phrases. BM25-style scoring balances term frequency, document frequency, and length normalization. Dense retrieval maps queries and passages into a vector space, recovering conceptual similarity and paraphrase. Neither dominates across all query types. Hybrid retrieval builds independent candidate lists and combines them.

Fuse ranks before comparing incompatible scores

Dense cosine scores and sparse scores do not share a calibrated scale. A raw weighted sum can be dominated by whichever retriever emits larger values. Reciprocal rank fusion (RRF) uses positions instead: for each document, add 1 / (c + rank) across result lists, where c dampens top-rank differences. Qdrant’s current Query API supports hybrid and multi-stage retrieval with prefetches and RRF/DBSF fusion; its documentation recommends tuning weighted RRF on held-out judgments rather than intuition. Elasticsearch likewise documents RRF as a hybrid-search fusion option.

# Illustrative, one-based ranks and c = 60
dense = {"policy-A": 1, "faq-B": 2, "policy-C": 3}
sparse = {"policy-C": 1, "policy-A": 2, "memo-D": 3}

def rrf_score(doc_id):
    ranks = [ranking[doc_id] for ranking in (dense, sparse)
             if doc_id in ranking]
    return sum(1 / (60 + rank) for rank in ranks)

# policy-A and policy-C gain support from both lists.

RRF is robust but discards score magnitude. Score-distribution normalization or learned fusion can exploit more information, but it needs validation and drift monitoring. Always retrieve deeper than the final k; fusion and reranking cannot select a document absent from every candidate list.

Rerank selectively

A cross-encoder or LLM reranker jointly inspects query and candidate and can resolve nuanced constraints. It adds cost and tail latency, and a reranker cannot repair missing candidates. Cache only when the query, corpus/index version, authorization scope, and reranker version make reuse safe. Test a cheap deterministic reranker or metadata boost as a baseline before adding another model call.

4. Vector geometry and approximate nearest neighbors

The similarity function must match model training and stored-vector treatment. Cosine compares direction; dot product combines direction and magnitude; Euclidean distance measures geometric separation. For unit-normalized vectors, ranking by cosine and dot product is equivalent, and squared Euclidean distance is monotonically related. Do not normalize reflexively if magnitude carries trained meaning. Record model, dimensions, preprocessing, normalization, and distance metric as one versioned contract.

Exact search is the quality oracle

An exact scan computes distances against all eligible vectors and provides the reference neighbor set for measuring approximate recall. It is often practical for small or tightly filtered subsets. Approximate nearest-neighbor (ANN) indexes trade perfect recall for lower latency and resource use. Keep an exact path in the benchmark environment; without it, you cannot tell whether missed results come from the embedding or the index.

HNSW mental model

Hierarchical Navigable Small World (HNSW) constructs a multi-layer proximity graph. Search enters a sparse upper layer, greedily approaches the query, and explores more neighbors in denser lower layers. m controls graph connectivity and therefore memory/build/search behavior. ef_construct expands the build-time candidate pool; ef_search (often surfaced as hnsw_ef or ef) expands the query-time search. Higher values commonly improve recall but cost build time, memory, or latency. Benchmark the actual filtered workload rather than repeating defaults.

Filtering changes the graph problem

A post-filter may leave too few candidates; a strict filter can also make graph traversal ineffective. Build indexes for frequent metadata filters and test selectivity slices. Qdrant documents a filterable HNSW approach and recommends creating payload indexes before ingestion so filter-aware graph edges can be built. In pgvector, approximate-index filtering is applied after the scan; its official README describes iterative scans as a way to search farther when filtering removes candidates. These implementation differences belong in a database decision and benchmark.

Quantization is an end-to-end trade

Quantization compresses vector representations to reduce memory and often accelerate distance work, at the cost of approximation error. Preserve original vectors when a two-stage search can rescore a larger compressed candidate set. Qdrant currently documents scalar, product, and binary approaches and explicitly frames the choice as accuracy, storage, and speed. Measure relevance, ANN recall against exact search, p50/p95/p99 latency, build time, and resident memory—not only compression ratio.

5. Benchmark relevance and performance together

A golden set contains representative queries plus graded or binary relevance judgments over evidence units. Sample navigational, exact-identifier, semantic, multi-hop, filtered, multilingual, fresh-content, long-document, and “no answer” cases. Split tuning from final validation so fusion weights and chunk sizes are not optimized on the score you report. Version queries, judgments, corpus snapshot, parsing, embedding, index configuration, and code.

Report macro averages and slices. A 2-point nDCG gain that hides a severe regression on one tenant, language, or exact-ID query is not a safe improvement. Inspect per-query deltas and categorize failures: parse loss, chunk boundary, stale index, ACL/filter error, candidate miss, fusion error, reranker error, packing loss, or generator misuse.

Measure under concurrency

Benchmark isolated stage latency and end-to-end latency. Warm and cold behavior differ. Include index build and freshness lag, throughput, error rate, CPU, memory, I/O, and cost per successful answer. Run controlled sweeps—candidate depth, ef, quantization, reranker depth—with fixed corpus and query set. Then load-test the best few configurations because tail latency can change under resource contention.

6. Qdrant as a production case study

Qdrant’s core data model is a collection of points, where a point has an ID, one or more vectors, and optional JSON payload. Named vectors allow different representations on the same point. Distance and dimensions are configured per vector. Payload fields support filtering; index frequent, security-relevant fields such as tenant or visibility before ingestion. The official collection documentation notes that point and indexed-vector counters can be approximate during optimization, so do not use them as an exact ingestion ledger.

Collection and tenancy choices

A collection per tenant gives strong operational separation but can create excessive collection/index overhead and complicate fleet-wide updates. A shared collection with tenant payload and a mandatory filter is efficient for many smaller tenants but makes authorization enforcement and noisy-neighbor testing critical. Dedicated collections may be appropriate for very large tenants, incompatible schemas, independent scaling, or embedding migrations. Put authorization-derived filters in trusted server code, not model output or user-provided query text.

Storage, index, and availability

Choose in-memory versus on-disk vectors and HNSW, quantization, and rescoring from the measured working set. Qdrant’s optimization guide documents configurations for low memory, high speed, and high precision; these are starting scenarios, not automatic recommendations. Sharding increases capacity and parallelism; replication improves availability and can increase read capacity, but multiplies storage and write work. Qdrant’s current distributed-deployment documentation notes that self-hosted shard balancing is operational work and recommends a load balancer so replicas and coordinators are not stranded behind one entry node.

Writes, backups, and migrations

Use deterministic point IDs and content hashes so reprocessing is idempotent. Write a source version into payload. Reconcile source-of-truth counts and hashes, not approximate index counters. Exercise snapshot creation and restore; a backup that has never restored is only an assumption. For an embedding change, build a new named vector or collection, backfill from the authoritative content, dual-read a shadow sample, compare quality and latency, switch an alias or routing layer, and retain rollback until freshness and parity checks pass. Qdrant provides current official guides for snapshots, migrations, and zero-downtime embedding-model migration; validate version-specific mechanics before execution.

7. Select, migrate, and debug the whole system

Choose a search engine by workload, not category labels. A dedicated vector system is attractive for vector-native filtering, multivector search, and independent scaling. Elasticsearch/OpenSearch can consolidate mature lexical search, aggregations, and hybrid retrieval. PostgreSQL with pgvector can minimize operational surface when transactional metadata and scale fit one system; pgvector offers exact search plus HNSW and IVFFlat, and its official project documentation is unusually explicit about filtered ANN trade-offs. Include team expertise, recovery, tenancy, write patterns, compliance, cost, and migration reversibility.

A safe migration sequence

1. Freeze and version the retrieval contract, judgments, and traffic sample.
2. Backfill the target from the authoritative source with deterministic IDs; reconcile counts and hashes.
3. Shadow reads and compare result overlap, relevance, filters, latency, and errors without affecting users.
4. Dual-write or capture a replayable change log; monitor freshness divergence.
5. Canary by tenant/query class, preserve old reads, and define automatic rollback gates.
6. Cut over, verify restore/DR and dashboards, then retire the old index only after the rollback window.

Search quality monitoring in production needs proxies plus sampled judgments. Track empty/low-confidence results, reformulations, citation clicks, answer abstentions, retrieval overlap by version, and user feedback. Do not equate click-through with relevance: position, presentation, and user urgency confound it. Convert investigated failures into the offline dataset.

Syllabus checkpoint: rewriting, deployment, and adjacent search

Separate query rewriting, expansion, and decomposition

Query rewriting produces a clearer search form of the same intent; expansion adds aliases, acronyms, entities, or lexical variants; decomposition produces separately retrievable subquestions. Preserve the original query and protected literals, version the transformation, cap fan-out, and compare each variant with an unchanged baseline. A rewrite that makes language fluent while dropping a product code, tenant, jurisdiction, negation, or date is a regression.

Deploy Qdrant on Kubernetes as a stateful system

A Kubernetes deployment needs persistent volumes, anti-affinity or topology spread, resource requests and limits, disruption planning, readiness behavior, secure credentials, snapshot/restore drills, and an upgrade/rollback runbook. Separate client-side retries from database replication, and test node loss while writes and searches continue. Stateful recovery and shard placement—not merely a green Pod—define readiness.

Do not omit Solr from migration discussions

Apache Solr remains relevant in Lucene-based enterprise search estates, especially where collections, shards/replicas, schema/analyzers, faceting, and SolrCloud operations already exist. A Solr migration inventory should capture analyzers, field types, synonyms, boosts, filters, faceting, update semantics, aliases, and operational SLAs. Do not translate only stored documents; translate relevance behavior and recovery procedures. Compare this with Elasticsearch/OpenSearch, Qdrant, and pgvector against the actual lexical/vector mix and operating constraints.

Interview playbook

For a retrieval design question, use RANKED:

1. R — Requirements: users, corpus, relevance definition, freshness, ACLs, scale, latency, and cost.
2. A — Authoritative data: source, parsing, stable IDs, versions, lineage, and deletion behavior.
3. N — Nomination: lexical/dense candidate generators, filters, depths, and exact baseline.
4. K — Keep order: fusion, reranking, deduplication, parent expansion, and packing.
5. E — Evaluate and expose: judgments, slices, metrics, traces, load tests, and error taxonomy.
6. D — Deploy safely: idempotent writes, sharding/replication, backups, canary, rollback, and SLOs.

Common traps

• Calling cosine similarity “accuracy,” or conflating ANN recall with relevance recall.
• Tuning on a few memorable queries and reporting the same queries as validation.
• Combining raw dense and sparse scores without calibration or normalization.
• Applying tenant filters after retrieval, allowing unauthorized candidates into context or traces.
• Assuming a larger chunk, larger k, or larger context window monotonically improves answers.
• Choosing a database from feature lists without measuring filtered workload, recovery, and team operations.
• Describing a migration as “reindex and switch” with no change capture, parity check, canary, or rollback.

Question bank

These prompts test retrieval reasoning, not memorized product vocabulary.

Proof artifact: a reproducible hybrid-search benchmark

Build a small, public-data search service that makes retrieval quality, filtered performance, and operational behavior inspectable. All results are portfolio measurements, not claims about Purnendu’s production experience.

Steps

1. Select a legally usable corpus with meaningful structure. Freeze a corpus manifest containing source URI, content hash, version, and parser result.
2. Create 150–300 queries with binary or graded evidence judgments. Include exact IDs, paraphrases, strict metadata filters, multiple required passages, stale versions, and no-answer cases. Split tuning and validation.
3. Implement four fixed variants: sparse baseline; dense exact/ANN; hybrid RRF; hybrid plus reranking. Keep parsing and corpus constant.
4. Sweep chunking, candidate depth, HNSW search effort, filter selectivity, and optional quantization. Record every configuration.
5. Run single-request and concurrent load profiles on declared hardware. Capture stage spans, p50/p95/p99, throughput, errors, CPU, memory, index size, and freshness lag.
6. Write a decision memo comparing Qdrant with an adjacent option such as pgvector or Elasticsearch for this workload, including a reversal condition.

Metrics

Report precision@5, recall@20, MRR, nDCG@10, evidence coverage, no-answer behavior, ANN recall against exact search, filtered-query slices, latency percentiles, index/build time, memory, and cost/resource assumptions. Show per-query deltas, confidence intervals or paired resampling when practical, and the error taxonomy—not only averages.

Deliberate failures

• Remove the payload index for a frequent strict filter and observe quality/latency under load.
• Lower ANN search effort until exact-neighbor recall visibly fails, then distinguish ANN loss from embedding relevance.
• Corrupt a parser boundary so an exception is separated from a policy statement; confirm the golden set catches it.
• Pause incremental indexing while source versions advance; ensure freshness monitoring and a user-visible policy respond.
• Make the reranker unavailable; verify timeout, fallback ranking, trace status, and bounded latency.

What to present

Present one pipeline diagram, the dataset card, a Pareto chart of nDCG versus p95 latency, two failure traces, and the one-page engine decision. Demonstrate one query where sparse wins, one where dense wins, one where a filter breaks naïve ANN, and one where the system correctly abstains.

Chapter review

Production retrieval is a chain of contracts. Preserve authoritative content and provenance, generate complementary candidates, fuse and rerank deliberately, pack evidence within a budget, and measure component as well as end-to-end behavior. Database parameters matter only in relation to a representative workload, relevance judgments, and operating constraints.

Glossary

ANN recall

The fraction of exact nearest neighbors recovered by an approximate index at a cutoff; distinct from judged relevance recall.

BM25

A lexical ranking family using term frequency, inverse document frequency, and document-length normalization.

Dense retrieval

Nearest-neighbor retrieval over learned vector representations.

HNSW

A hierarchical proximity-graph ANN structure with build, memory, latency, and recall trade-offs.

Payload index

An index over metadata used to accelerate or integrate filtering with retrieval.

RRF

Reciprocal rank fusion, which combines result lists using document positions rather than raw score scales.

Reranker

A later-stage scorer that evaluates a query and candidate more jointly than a first-stage retriever.

Evidence coverage

Whether packed context contains all facts required to answer a task, not merely one relevant chunk.

Mastery checklist

• I can isolate ingestion, candidate, rank, pack, and generation failures.
• I can give a query where sparse wins and one where dense wins.
• I can calculate a simple RRF result and explain when it is insufficient.
• I can separate embedding relevance, ANN recall, and end-to-end answer quality.
• I can explain how HNSW parameters and strict filters change the workload.
• I can define precision, recall, MRR, and nDCG and choose among them.
• I can design a tenant-safe, versioned incremental index.
• I can outline a shadowed, canaried, reversible search migration.
• I can defend an engine choice using workload evidence rather than feature slogans.

1. Earn the right to be agentic

An agent is a system in which a model chooses at least part of the action sequence at runtime. That flexibility is useful when the task is open-ended, the correct path depends on observations, and tool selection cannot be exhaustively encoded. It also expands the state space: more trajectories, model calls, permissions, latency, cost, and failure combinations. “Agentic” is therefore a design choice, not a maturity level.

Apply the uncertainty–consequence test. Agent value rises with path uncertainty: research, diagnosis, codebase exploration, or heterogeneous support requests. Required control rises with consequence: moving money, deleting data, changing production, contacting a customer, or disclosing sensitive information. High uncertainty plus high consequence calls for a bounded agent that prepares evidence and a plan, then an explicit approval before action.

Define success and terminal outcomes

“Helpful response” is not an operational contract. Define allowed terminal states such as completed, needs_user_input, awaiting_approval, blocked_by_policy, budget_exhausted, and dependency_failed. For each, specify what the user sees and whether resumption is possible. A loop without a terminal-state model is a reliability bug waiting for traffic.

2. Choose an orchestration pattern from the control problem

Framework names matter less than who decides the next step and where state lives. Model the workflow as states, events, guarded transitions, side effects, and terminal states. Then choose a framework—or plain code—that expresses this model clearly.

Worked example: bounded refund investigation

Consider a support workflow that may inspect an order and draft a refund, but cannot issue it without policy checks and approval above a threshold. A useful graph is:

ALLOWED = {
    "validated": {"looked_up", "needs_user_input"},
    "looked_up": {"proposed", "not_found"},
    "proposed": {"policy_denied", "awaiting_approval", "approved"},
    "awaiting_approval": {"approved", "rejected"},
    "approved": {"completed", "dependency_failed"},
}

def transition(state, next_status):
    if next_status not in ALLOWED.get(state["status"], set()):
        raise ValueError("invalid workflow transition")
    return {**state, "status": next_status}

The model may propose an action and rationale, but deterministic code verifies policy and authorization. The approval stores the exact proposed action, resource ID, amount, policy version, and expiry. Execution uses an idempotency key. The system never interprets approval as permission for a later, altered action.

3. Treat tool calls as untrusted requests

Function calling is a protocol round trip: the application describes tools; the model emits a structured request; application code validates and authorizes it; the application executes the operation; and a structured result returns to the model. Both OpenAI and Anthropic document this separation: the model proposes arguments, while client-side application code performs the function. Never let the model’s selection bypass normal service controls.

A strong tool contract

• Narrow intent: get_order_status is safer and easier to select than run_api_request.
• Constrained schema: use enums, bounded lengths/ranges, required fields, and reject unknown fields where supported.
• Trusted identity: derive tenant, actor, and scopes from authenticated context; do not accept them as model-controlled arguments.
• Clear effects: distinguish read, reversible write, irreversible write, and external communication.
• Idempotency: mutation tools accept or derive a key tied to workflow and semantic operation.
• Typed result: separate ok, retryable error, permanent error, policy denial, and not-found; keep user-safe and operator detail distinct.
• Limits: server-enforced timeout, pagination, result-size cap, rate/quota, and redaction.

{
  "name": "prepare_refund",
  "description": "Create a reviewable refund proposal; does not issue funds.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {"type": "string", "pattern": "^ord_[A-Za-z0-9]+$"},
      "reason": {"type": "string", "maxLength": 500},
      "amount_minor": {"type": "integer", "minimum": 1}
    },
    "required": ["order_id", "reason", "amount_minor"],
    "additionalProperties": false
  }
}

Structured-output support can guarantee or improve schema conformance depending on provider and mode, but schema validity is not semantic validity. A perfectly formed order ID can belong to another tenant; a valid amount can exceed the refundable balance. Revalidate business invariants at execution time. Keep tool descriptions accurate and treat third-party descriptions as untrusted metadata.

Return errors the orchestrator can act on

A timeout, invalid input, expired credential, policy denial, and missing record need different control flow. Avoid returning a prose blob that the model must reinterpret. Use stable error codes, retryability, safe user message, and correlation ID. Do not expose tokens, stack traces, raw database errors, or private tool results to the model unnecessarily.

4. Design for interruption and replay

Multi-step systems fail between steps. A process can crash after an external write but before recording success; a user can approve hours later; a provider can time out after completing a request. Durable execution requires explicit persisted state and replay-safe side effects—not merely “retry three times.”

Checkpoint at meaningful boundaries

Persist input references, validated state, chosen route, model/prompt/tool versions, tool request and result references, budget consumption, approvals, and terminal status. Store only the content required for recovery and audit; encrypt or reference sensitive payloads and apply retention. LangGraph’s official persistence documentation describes thread-scoped checkpoints, pending writes, fault recovery, and time-travel debugging. Its interrupt documentation notes that state is saved for human-in-the-loop resumption; side effects before an interrupt must be idempotent because the node can replay.

Retry ownership

Place retries at one layer whenever possible. SDK, HTTP client, orchestrator, queue, and tool service each retrying can multiply calls. Retry only transient failures, with exponential backoff, jitter, a deadline, and a total attempt budget. Respect provider retry hints. Do not retry policy denials or deterministic validation failures. If a request has ambiguous outcome, reconcile using an operation key or read-before-retry.

Termination is a product feature

Enforce maximum wall time, model calls, tool calls, repeated identical calls, replans, tokens, and monetary budget. Detect no-progress cycles using normalized action/result fingerprints. On exhaustion, preserve a partial result and missing requirements where safe. A hard “something went wrong” after ten hidden retries wastes both evidence and trust.

5. Separate context, state, and memory

Conversation history is not a database, and a vector store is not automatically memory. Use three distinct concepts:

• Working context: bounded messages, evidence, tool results, and instructions needed for the current model call.
• Workflow state: authoritative typed fields required to resume and enforce transitions.
• Long-term memory: intentionally retained facts or summaries available across sessions, with provenance, consent, correction, and deletion.

Keep raw authoritative values in workflow state; format them into prompts at call time. Summaries are lossy and should carry source references and version. Treat retrieved memory as untrusted context, because it can be stale, incorrectly attributed, or poisoned. Enforce tenant/user boundaries before retrieval. Do not store secrets or sensitive tool results merely because they might help a later response.

Manage the context window as a budget

Reserve space for system/tool schemas, current request, evidence, and output. Drop irrelevant history; summarize older turns with explicit unresolved commitments; retrieve only task-relevant memory; and cap tool outputs. Compaction can lose an instruction or pending constraint, so evaluate long-running conversations and resume cases. Cache stable prefixes only when provider semantics, privacy, and version invalidation are understood.

Models and prompts are policy versions

Route by task risk and measured capability, not prestige. A small model may classify or extract; a stronger model may plan or resolve ambiguity; deterministic code validates and executes. Fallback is not simply “another model”: providers differ in schema subsets, tool-call formats, stop reasons, safety behavior, and context handling. Create a provider adapter and run the same contract/evaluation suite per route. Version system prompt, tool catalog, schemas, model snapshot, temperature/effort settings, and retrieval configuration with every trace.

6. Put authority outside the model

Prompt injection is a control-flow attack: untrusted content attempts to redefine instructions or induce tool use. Label and delimit external content, but do not depend on prompting alone. Authorization must be enforced by trusted code at the tool boundary. Give each workflow an explicit capability set and derive scopes from the authenticated actor, tenant, environment, and approved purpose.

Approval design

Request approval for the exact action, not a vague plan. Show target, effect, sensitive fields, cost/amount, environment, and why the action is requested. Bind approval to a content hash, policy version, actor, and expiry; invalidate it when material arguments change. Separate proposer from executor where risk warrants. Log the decision and make rejection a first-class state that can include feedback.

MCP is interoperability, not an authorization shortcut

The Model Context Protocol standardizes connections among hosts, clients, and servers and exposes resources, prompts, and tools. The current specification explicitly treats tool execution and arbitrary data access as high-risk and emphasizes consent and control. Its authorization specification requires resource-bound tokens for HTTP flows and forbids token passthrough. An MCP server still needs authentication, per-tool scopes, tenant isolation, input validation, output redaction, rate limits, audit logs, and downstream credentials distinct from inbound credentials.

7. Operate the agent as a distributed system

One user request may span router, model, retriever, several tools, approval wait, and final synthesis. Give it a trace ID and create spans for each meaningful step. Capture workflow/step name, model and prompt version, tool name, attempt, status, latency, token usage, candidate/argument size, cache outcome, budget remaining, and safe error category. Do not record raw prompts, tool inputs, or outputs by default; OpenTelemetry’s GenAI attribute documentation warns that message content can contain sensitive information.

Observe outcomes and trajectories

End-to-end task success alone hides inefficient or unsafe routes. Measure correct tool selection, argument validity, authorization denials, unnecessary calls, repeated calls, plan changes, step success, approval rate/time, recovery success, and terminal-state distribution. Pair quality with latency and cost. Sample failed, expensive, long, denied, and novel traces into evaluation datasets.

Use hierarchical budgets

Start with an end-to-end deadline and cost ceiling. Allocate child timeouts per dependency with room for response construction. Enforce model-call, tool-call, parallel-branch, output-token, and retry budgets. Cancel abandoned work when the client disconnects or the outcome becomes terminal. Streaming improves perceived latency but complicates error semantics: distinguish provisional progress from committed result, and never stream a claim of success before a side effect is confirmed.

Degrade by preserving the user’s goal

• If the planner fails, fall back to a fixed supported workflow or ask a targeted question.
• If a nonessential tool fails, return a partial result with the missing source named.
• If the primary model is unavailable, use a validated fallback only for routes it passed.
• If approval infrastructure is unavailable, pause; do not silently auto-approve or discard the proposal.
• If a mutating tool has ambiguous outcome, reconcile before telling the user to retry.

Syllabus checkpoint: API contracts, memory, routing, and model adaptation

Tool calling or function calling is a typed proposal from a model, not authorization to execute. Validate structured output against a schema, enforce policy in code, convert provider-specific tool messages into an internal contract, and test malformed arguments, unknown tools, duplicate calls, timeouts, and partial streaming failures. OpenAI, Anthropic, and Gemini expose different request/response details; the durable design is a narrow internal tool interface plus versioned adapters.

Short-term and long-term memory are different data products

Short-term memory keeps the current task coherent—recent messages, tool results, and summarized state—within an explicit token and privacy budget. Long-term memory retrieves durable facts or prior outcomes and therefore needs provenance, tenant isolation, retention/deletion, conflict handling, and a rule for when old memory is untrusted. Do not silently promote a model inference into a user fact.

Model routing and fallback

Route only on observable requirements such as modality, context size, risk class, tool reliability, latency target, or evaluated task performance. A fallback must preserve the contract: schema, safety policy, tool permissions, and user-visible limitations. Record route, model/version, trigger, tokens, latency, cost, and result so the policy can be evaluated rather than becoming invisible complexity.

Fine-tuning and distillation decision criteria

Prefer prompting, retrieval, deterministic rules, or tool design when the problem is missing knowledge, fresh data, or unsafe authority. Consider fine-tuning when a stable, well-labeled behavior or format repeatedly resists prompting and the evaluation set proves the gap. Consider distillation when a capable teacher produces a large, quality-controlled dataset and a smaller model can meet a measured quality threshold with better latency or cost. Include data rights, privacy, training/evaluation leakage, maintenance, rollback, and provider portability in the decision.

Interview playbook

Use AGENTS to structure a design answer:

1. A — Aim and authority: user outcome, risk, actor, tenant, and actions the system may never take.
2. G — Graph and state: deterministic baseline, model decisions, transitions, invariants, terminal states.
3. E — Execution contracts: narrow tools, schemas, validation, idempotency, typed results, deadlines.
4. N — Non-happy paths: retries, ambiguous writes, dependency failure, no progress, rejection, resumption.
5. T — Telemetry and tests: traces, versions, trajectory/outcome evals, safety cases, release gates.
6. S — Spend and safe rollout: token/tool/time budgets, shadow mode, approvals, canary, fallback.

Common traps

• Calling a prompt chain an “agent” without identifying any runtime decision.
• Letting model-generated tenant IDs, URLs, SQL, or scopes reach a tool unvalidated.
• Assuming structured output means business-correct or authorized output.
• Retrying a timed-out mutation without idempotency or reconciliation.
• Using conversation history as authoritative workflow state.
• Adding multiple agents when independent tools or parallel functions would suffice.
• Describing human approval without binding it to exact, expiring action arguments.
• Tracing sensitive content by default or omitting model/prompt/tool versions.

Question bank

Answer with control boundaries, measurable failure behavior, and the smallest justified architecture.

Proof artifact: a resumable approval-bound agent

Build a support workflow against a fake order service. It may read an order, retrieve a policy, propose a refund, and—only after deterministic policy checks—execute a low-value simulated refund or request human approval. Use synthetic data and fake funds. The artifact demonstrates controls, not production outcomes.

Steps

1. Write the authority matrix and terminal states first. Mark which transitions are deterministic, model-selected, or human-controlled.
2. Implement typed state and narrow read/proposal/execution tools. Inject actor and tenant server-side; add idempotency keys to execution.
3. Persist checkpoints, version prompt/model/tool schemas, and support resume after process restart.
4. Add end-to-end traces with safe metadata, per-step timing/tokens, attempts, budget, approval, and terminal reason.
5. Create an evaluation set for route choice, argument accuracy, policy result, trajectory length, final outcome, injection resistance, and user-visible recovery.
6. Run in shadow mode over synthetic scenarios, then enable only the reversible fake action behind a feature flag.

Metrics

• Task completion and correct terminal-state rate by scenario.
• Tool selection precision/recall, argument validity, and business-invariant pass rate.
• Unauthorized action attempts and cross-tenant disclosures—both must be zero in the test suite.
• Median/p95 model calls, tool calls, tokens, wall time, and example cost per terminal state.
• Duplicate side effects after replay—must be zero with idempotent fake execution.
• Checkpoint recovery and user-visible safe-degradation success rate.

Deliberate failure injection

Crash after the fake refund service commits but before the workflow records success; resume and prove no duplicate. Time out policy retrieval; exhaust the model-call budget; change a proposal after approval and verify approval invalidation; return malicious instructions inside a policy document; submit a model-generated tenant ID; and make the approval service unavailable. Capture trace and terminal behavior for each.

What to present

Present the state diagram, authority matrix, one tool schema, a crash-and-resume trace, an injection-denial trace, evaluation results by slice, and a short argument for which steps were deliberately kept non-agentic. Label every metric as a synthetic artifact measurement.

Chapter review

Reliable agent engineering is control engineering around probabilistic decisions. Keep authority in trusted code, express state and termination explicitly, make tools narrow and replay-safe, persist meaningful checkpoints, bound every resource, and evaluate both outcome and trajectory. Add autonomy only where runtime uncertainty creates measured value.

Glossary

Agent

A system in which a model chooses part of the action sequence at runtime.

Capability

An explicitly granted operation available to a workflow, distinct from what a model requests.

Checkpoint

A persisted workflow boundary from which execution can be inspected or safely resumed.

Idempotency key

A stable operation identifier that lets a service return the same semantic result without repeating the effect.

Interrupt

A deliberate pause that persists state and awaits external input such as approval.

Trajectory

The ordered sequence of model decisions, tool calls, observations, and transitions.

Working context

The bounded information supplied to one model call, not an authoritative state store.

MCP

Model Context Protocol, a standard for connecting hosts/clients with servers exposing resources, prompts, and tools.

Mastery checklist

• I can justify every model-selected step against a deterministic baseline.
• I can draw states, guarded transitions, terminal outcomes, and recovery paths.
• I can design a tool whose schema, authority, effect, and errors are explicit.
• I can explain ambiguous mutation outcomes and idempotent recovery.
• I can distinguish working context, workflow state, and long-term memory.
• I can bind human approval to one exact, expiring action.
• I can explain MCP’s integration value without outsourcing security to the protocol.
• I can define trace fields, privacy controls, and hierarchical budgets.
• I can evaluate trajectory quality as well as final task success.

1. Quality is a decision system, not a score

An evaluation system exists to make decisions: continue iterating, merge a change, canary it, expand rollout, roll back, investigate a slice, or escalate a risk. Start by writing the decision and the consequence of a false pass or false fail. Only then choose metrics and thresholds. A single “quality score” cannot represent correctness, evidence use, safety, latency, cost, and user value without hiding important trade-offs.

Build a quality tree

For an enterprise policy assistant, the top outcome might be “authorized users resolve policy questions accurately and quickly.” Decompose it into: relevant current evidence retrieved; answer claims supported; citations resolvable; uncertainty handled; unauthorized information never disclosed; correct escalation on ambiguous or high-risk questions; and acceptable latency/cost. Attach at least one measure and one failure example to each leaf.

2. Engineer the dataset before the evaluator

A golden set is a versioned collection of inputs, expected properties, metadata, and judgments. “Golden” means reviewed, traceable, and stable enough for comparison—not perfect or frozen. OpenAI recommends task-specific tests that reflect real distributions and continued human calibration; LangSmith and Langfuse both support offline-to-production feedback loops.

Four complementary sources

1. Curated core: expert-written canonical and boundary cases.
2. Production traces: privacy-reviewed samples of common, failed, costly, uncertain, and novel interactions.
3. Adversarial cases: injection, leakage, malformed input, unavailable tools, contradictory sources, and no-answer examples.
4. Synthetic expansion: reviewed, provenance-labeled paraphrases or rare combinations; supplements, not proof of representativeness.

Each record should include stable ID, input, reference evidence or expected behavior, rubric, risk, source/provenance, created/reviewed dates, language, tenant/data class, intent, difficulty, and applicable evaluators. Preserve an immutable raw trace reference separately when allowed. Redact or synthesize sensitive values before placing cases in developer-visible stores.

Split by how data can leak

Keep development examples for iteration, validation for thresholds, and a sequestered test set for final claims. Group by source document, user/thread, template, or time so near-duplicates do not cross splits. Repeated optimization makes validation data de facto training data; preserve a final untouched set.

Version everything that changes meaning

Version dataset, judgments, rubrics, corpus/parser/retrieval, prompt/model/tools, evaluators, sampling, and environment. Store hashes and a changelog. A score without dataset and evaluator versions is not reproducible evidence.

3. Match metrics to the failure boundary

Component scores localize defects; end-to-end scores reveal interactions. Retrieval recall can rise while excess context harms answers; correct tool selection can still carry invalid arguments.

Retrieval and RAG

• Retrieval: precision@k, recall@k, MRR, nDCG, filtered slices, and evidence coverage after packing.
• Grounding: claim-level support by retrieved evidence, citation correctness, citation completeness, and contradiction.
• Answer: task correctness/completeness, relevance, instruction adherence, appropriate uncertainty, and abstention.

Faithfulness and correctness differ: an answer can repeat stale evidence faithfully or be correct yet unsupported. Evaluate both. Ragas exposes RAG and agent metrics, but inspect definitions, prompts, model dependence, and intermediate outputs before using any as a gate.

Agents and tool use

• Goal/task completion and correct terminal state.
• Tool-selection precision/recall and unnecessary-call rate.
• Argument schema and semantic validity; authorization/policy result.
• Trajectory efficiency: calls, retries, replans, repeated steps, tokens, and wall time.
• Side-effect correctness, duplicate effects, recovery, and escalation.

Deterministic output checks first

Use exact match for known classifications, JSON Schema for structure, parsers/compilers for code or queries, executable tests for calculations, database comparisons for extraction, and policy engines for allowed actions. A deterministic checker is cheaper, faster, repeatable, and easier to debug than a model judge when the property is mechanically decidable.

4. Calibrate human and model judgment

Human review is nuanced but slow and variable. Model judges scale but inherit prompt sensitivity, bias, position/verbosity preferences, and domain blind spots. Prefer deterministic checks, use humans for ground truth/high-risk ambiguity, and calibrate model judges for broader coverage.

Write an operational rubric

Replace “good answer, 1–5” with observable anchors. For faithfulness, label material claims supported, contradicted, or absent from evidence. Define required and harmful task behavior, give boundary examples, and permit “insufficient information.”

Measure annotator behavior

Train reviewers on shared cases, blind variant identity, randomize order, and double-label a stratified subset. Track agreement by criterion/slice and adjudicate. Low agreement can expose training, rubric, evidence, or unsettled-requirement problems.

Calibrate an LLM judge

1. Build a held-out, human-labeled calibration set with hard and boundary cases.
2. Give the judge only the information the rubric requires; prevent candidate metadata from revealing the variant.
3. Prefer categorical or pairwise decisions with explicit anchors when absolute numeric scoring is unstable.
4. Test candidate-order reversal, verbosity, self-preference/provider, reference leakage, and prompt-injection inside the content being graded.
5. Measure agreement, confusion matrix, false-pass rate on high-risk cases, and stability across repeated runs.
6. Version judge model, prompt, temperature/settings, rubric, and calibration result; recalibrate after change.

Do not let the judge’s explanation substitute for correctness. Store reasoning as debugging material, not proof. If a judge is weak on a critical slice, route that slice to a deterministic check or human review. For release gates, prioritize the false-pass cost: an evaluator that misses unsafe behavior is worse than one that occasionally sends a safe run for review.

5. Turn failures into an error taxonomy

Aggregate scores show population movement; error analysis chooses the repair. Give each failure a primary stage, symptom, likely cause, consequence, and owner. Secondary tags can capture interactions.

Slice before celebrating

Choose slices from risk and plausible causes: intent, language, policy regime, tenant/data class, document type, exact identifier, multi-hop, no-answer, tool, route, cohort, and age. Report counts and uncertainty; define critical slices before the experiment.

Inspect paired deltas

List paired improvements and regressions. Equal means can hide replacing harmless style errors with one security failure. Inspect largest negative deltas and every invariant violation; track severity and consequence.

6. Build release gates that tolerate variability, not regressions

CI compares immutable baseline and candidate configurations on a pinned dataset/evaluator suite. Record environment and inspectable responses when policy permits. Invalidate caches for every changed prompt, model, retrieval, or tool dimension.

def release_decision(base, candidate):
    hard_fail = any(candidate[name] != 1.0 for name in (
        "tenant_isolation", "schema_valid", "no_duplicate_effect"
    ))
    quality_drop = candidate["task_success"] < base["task_success"] - 0.02
    slow = candidate["p95_ms"] > 1.10 * base["p95_ms"]
    costly = candidate["cost_per_success"] > 1.15 * base["cost_per_success"]
    return "block" if hard_fail or quality_drop or slow or costly else "canary"

# Thresholds above are illustrative; derive real gates from product risk.

Hard invariants require every applicable case to pass. Comparative metrics need both absolute floors and allowable deltas. Critical slices need their own gates. Cost and latency are guardrails. Treat missing evaluator output as a failure or explicit “inconclusive,” never as a pass. Keep a small smoke suite on every change and a larger suite on scheduled runs or release candidates, while ensuring high-risk cases remain in the fast gate.

Account for stochastic and sampling uncertainty

Use paired comparisons and report intervals via paired bootstrap or another appropriate method; inspect discordant binary outcomes. Repeat a stratified subset to estimate variance. Never average away safety failures, and weigh practical—not only statistical—significance.

Move online carefully

Shadow, then canary or A/B test using a randomization unit that avoids contamination. Predefine outcome, guardrails, exposure, stop rules, and analysis; monitor assignment and segment regressions. Never expose an unsafe variant for statistical power.

7. Close the offline–online quality loop

Offline datasets provide controlled repeatability; production provides distribution reality. Instrument traces with application, prompt, model, retrieval, tool, evaluator, and release versions plus safe outcome metadata. Sample common traffic randomly for prevalence, and oversample rare/high-risk signals for discovery. Keep weighting clear: a risk-enriched review queue cannot estimate population quality without correcting its sampling design.

Production signals are evidence, not ground truth

Track feedback, completion, abandonment, reformulation, escalation, citations, corrections, latency, cost, and denials. Each is confounded: clicks reflect position and silence may mean abandonment. Calibrate proxies with reviewed traces.

Choose tools by data model and exit path

Keep manifests and core evaluators portable. Export scores, case/trace references, and versions. OpenTelemetry semantics evolve and content can be sensitive; pin conventions and avoid message bodies by default.

8. Make safety and privacy executable

Build adversarial cases from actual inputs: user text, retrieval, tools, files, connectors, memory, and tenant data. Use the NIST Generative AI Profile to structure governance, then translate it into system-specific controls and tests.

Test the full threat path

• Prompt injection: direct user instructions and indirect instructions embedded in retrieved pages, documents, tool output, or memory.
• Sensitive information: secrets, PII, hidden prompts, credentials, and private records requested directly or inferred through side channels.
• Tenant isolation: IDs from another tenant, mixed-index candidates, cached responses, trace views, and shared memory.
• Unsafe tool use: unauthorized tool, excessive scope, altered action after approval, malicious URL/arguments, duplicate effect, and ambiguous timeout.
• Policy behavior: refusal consistency, over-refusal on benign requests, safe alternatives, and correct human escalation.
• Robustness: malformed encoding, extreme length, empty/contradictory evidence, unavailable dependencies, and partial streaming.

Measure attack success rate, sensitive-data disclosure, unauthorized-action rate, false refusal, escalation precision/recall, time to detection, and recovery behavior by attack surface. Hard security boundaries should be enforced deterministically and must pass every applicable test. Model-based red teaming can expand cases, but manually validate coverage and preserve sequestered attacks to reduce test gaming.

Protect the evaluation system itself

Datasets and traces contain revealing failures. Apply minimization, access control, encryption, retention/deletion, tenant partitioning, and audit. Treat candidate text as untrusted judge input; sandbox code evaluators with minimal permissions. Send production content externally only under an approved data contract.

Syllabus checkpoint: metric language and evaluation tooling

Faithfulness, groundedness, and relevance

Teams use these labels inconsistently, so define the rubric before quoting a score. A practical convention is: faithfulness asks whether answer claims are supported by supplied evidence; groundedness asks whether the answer is anchored to the authorized source context rather than unsupported model knowledge; relevance asks whether the response addresses the user’s actual request. Add completeness and citation correctness separately—a faithful answer can still omit the decisive exception.

Deterministic checks, probabilistic scoring, and human annotation

Use deterministic checks for schemas, required fields, citations that resolve, forbidden strings, permissions, exact calculations, and known invariants. Use probabilistic scoring for semantics that tolerate legitimate variation, but calibrate thresholds and uncertainty. Human annotation needs an operational rubric, examples near decision boundaries, blinded ordering where possible, adjudication, agreement measurement, and recorded annotator context. Sampling must represent important slices rather than only easy or recently failed cases.

Choose a platform by the evaluation loop, not the dashboard

LangSmith and Langfuse can manage traces, datasets, experiments, and feedback with different hosting and ecosystem trade-offs. RAGAS and DeepEval provide useful evaluation building blocks; custom Python harnesses remain valuable for exact product contracts. Arize Phoenix and commercial Arize capabilities add tracing/evaluation and observability workflows. OpenTelemetry supplies vendor-neutral trace context and export. Regardless of tool, require dataset, prompt, model, judge, code, and environment versions plus links from aggregate regressions to inspectable traces.

Interview playbook

Use QUALITY to answer an evaluation-system design prompt:

1. Q — Question and consequence: Which release/product decision, and what does a false pass cost?
2. U — User distribution: traffic, important slices, risks, and no-answer/edge behavior.
3. A — Artifacts and annotations: dataset sources, provenance, rubric, splits, versions, and privacy.
4. L — Layers of measures: deterministic, component, end-to-end, human, judge, safety, and operations.
5. I — Inspect errors: paired deltas, taxonomy, severity, slices, uncertainty, and evaluator failures.
6. T — Threshold and trial: hard invariants, regression gates, shadow/canary/A-B design, rollback.
7. Y — Yield feedback: production sampling, adjudication, new cases, ownership, and change cadence.

Common traps

• Choosing metrics before defining the product decision and failure cost.
• Using one aggregate judge score with no rubric, calibration, or slice analysis.
• Calling synthetic questions representative without validation against production.
• Tuning prompt, threshold, and evaluator on the same test set and reporting it as generalization.
• Allowing quality gains to compensate mathematically for security or privacy violations.
• Comparing unpaired runs with changed corpus, parser, environment, and model all at once.
• Treating user feedback, clicks, or judge explanations as uncontested ground truth.
• Building dashboards without an owner, alert/action threshold, or rollback path.

Question bank

These questions test whether evaluation evidence can support a production release decision.

Proof artifact: a versioned RAG release gate

Build a local, vendor-neutral evaluation harness for the public-data RAG system from chapter 2. It should compare baseline and candidate, publish case-level diffs, and return a failing process status when a release gate is violated. All thresholds and results are portfolio examples, not claims about Purnendu’s work history.

Steps

1. Create a JSONL or table-based dataset with stable IDs, input, reference evidence, expected properties, slice tags, risk, provenance, and split. Hash the manifest.
2. Run a frozen baseline and one candidate on the same validation cases. Capture retrieval IDs, packed context, response, citations, tool/trace events, latency, tokens, cost estimate, and all versions.
3. Implement deterministic schema, citation-resolution, tenant-isolation, and no-duplicate-effect checks; retrieval metrics; and a human-calibrated claim-support judge.
4. Generate overall and slice tables, paired deltas, error taxonomy, invariant failures, and Pareto plots. Persist evaluator failures separately.
5. Encode release rules: all hard invariants pass, minimum quality floor, maximum allowable regression overall and on critical slices, and latency/cost budgets.
6. Run the small critical suite in CI. Schedule the full suite and require the report for canary promotion. Document owner and rollback trigger.

Metrics

Capture recall@20, nDCG@10, packed evidence coverage, claim support, task success, citation correctness, abstention, safety invariants, evaluator agreement on a calibration subset, p50/p95/p99 latency, tokens, cost per successful task, and failures by taxonomy/slice. Include sample counts and uncertainty.

Deliberate failure injection

• Insert one cross-tenant evidence item and prove the invariant blocks the release even if average relevance rises.
• Alter the judge prompt to favor verbose answers and demonstrate calibration/order tests detect drift.
• Remove citations after generation and verify deterministic resolution/completeness checks fail.
• Slow the reranker and confirm the latency guardrail catches the p95 regression.
• Leak near-duplicate source questions across splits, observe the inflated score, then repair grouping and document the change.

What to present

Show the quality tree, dataset card/manifest hash, evaluator calibration matrix, baseline-versus-candidate slice report, one blocked CI run, two error traces, and the exact release decision. The strongest demonstration is a candidate with a better aggregate score that the gate correctly refuses because a critical invariant or slice regressed.

Chapter review

Evaluation engineering turns variable model behavior into bounded release decisions. It begins with product consequences and representative data, uses the cheapest valid evaluator at each boundary, calibrates human and model judgment, inspects errors and slices, and connects offline evidence to guarded production rollout. The evaluation system itself is versioned, tested, monitored, and protected.

Glossary

Golden set

A reviewed, versioned dataset with inputs, provenance, expected properties/evidence, metadata, and judgments for repeatable comparison.

Guardrail metric

A limit that an optimization may not violate, such as p95 latency or cost per successful task.

Hard invariant

A property that must always hold, such as tenant isolation; it is not averaged with softer quality metrics.

LLM-as-judge

A model-driven evaluator that applies a rubric to an output or comparison and must be calibrated like any other model component.

Paired evaluation

Comparison of variants on the same cases, enabling per-case deltas and more efficient uncertainty analysis.

Slice

A meaningful subset defined by risk, user group, intent, data, or causal mechanism.

Faithfulness

Whether response claims are supported by supplied evidence; distinct from real-world correctness.

Evaluation leakage

When test information influences development or related examples cross splits, inflating apparent generalization.

Mastery checklist

• I can draw a product-specific quality tree and label objectives, guardrails, and invariants.
• I can design dataset sources, provenance, grouped splits, slices, and a version manifest.
• I can distinguish retrieval, packing, grounding, answer, agent, and operational metrics.
• I choose deterministic evaluators whenever the property is mechanically decidable.
• I can calibrate a model judge against blinded human labels and test its biases.
• I can turn case failures into an actionable taxonomy and paired slice report.
• I can specify CI gates, uncertainty handling, shadow/canary, and rollback.
• I can design injection, leakage, tenant, tool, refusal, and escalation tests.
• I can compare evaluation platforms while preserving portability and data governance.

1. Treat every integration as a changing contract

An enterprise connector is a small distributed system at an organizational boundary. The remote team controls its schema, quotas, release cadence, and incident response; your team owns the consequences. Start by writing the contract you need, then isolate the vendor-specific adapter behind it.

REST, GraphQL, and the canonical model

REST often gives straightforward resource endpoints, caching semantics, and operational visibility. GraphQL can reduce over-fetching and combine related reads, but query complexity, pagination conventions, partial errors, and field-level authorization still need explicit handling. Neither protocol removes the need for a canonical internal model. Convert remote employees, accounts, or opportunities into stable internal types at the edge so provider renames do not spread throughout the product.

Design the public API around jobs

A sync that may take minutes should not hold an HTTP connection open. Accept a request, validate authorization and idempotency, persist a job, enqueue it, and return 202 Accepted with a durable status URL. The status resource should expose a state machine such as queued → running → succeeded | partially_succeeded | failed | cancelled, counts, timestamps, a sanitized error summary, and links to audit details. Cancellation is a request, not an instant fact: workers must observe it at safe checkpoints.

POST /v1/tenants/{tenant_id}/sync-jobs
Idempotency-Key: 7f8f...              # scoped to tenant + operation

202 Accepted
{
  "job_id": "job_01...",
  "status": "queued",
  "status_url": "/v1/sync-jobs/job_01..."
}

For pagination, prefer an opaque cursor tied to a deterministic sort key. For compatibility, use consumer-driven contract tests, tolerant readers for additive fields, and a deprecation process with telemetry showing which clients still use an old version. “We will version later” is not a strategy.

2. Identity, tenant boundaries, and webhook admission

Authentication proves who is calling; authorization decides what that identity may do; tenant routing decides whose data the action can touch. Keep all three visible in the design.

Choose the principal deliberately

The current IETF OAuth security best practice recommends authorization code flows with PKCE, exact redirect-URI matching, protection from mix-up and CSRF attacks, and sender-constrained tokens where applicable; it also deprecates insecure legacy patterns. Use this as the baseline rather than treating the original OAuth 2.0 RFC as the last word: RFC 9700.

Webhook admission pipeline

1. Read the exact raw bytes with a strict size and content-type limit. Do not parse and re-serialize before verification.
2. Select the secret by authenticated endpoint or connection identifier, not by a tenant ID trusted from the payload.
3. Verify the provider’s HMAC or asymmetric signature using a constant-time comparison. GitHub’s current guidance, for example, signs the payload and recommends secure comparison: validating webhook deliveries.
4. If the protocol supplies a signed timestamp, enforce a bounded age and record the delivery ID to resist replay. Signature validity alone does not prove freshness.
5. Insert the raw event metadata and payload hash into an inbox table with a unique key, acknowledge quickly, and process asynchronously.

During secret rotation, accept the old and new secret for a short, audited overlap, but keep the accepted key version on the delivery record. Never log bearer tokens, webhook secrets, full sensitive payloads, or unredacted model prompts. A correlation ID is useful; a credential is not.

3. Delivery semantics are end-to-end properties

Brokers describe transport behavior; the business outcome also depends on producers, consumers, databases, and external side effects. Say exactly where duplication or loss can occur.

At-most-once, at-least-once, and the exactly-once boundary

• At-most-once: do not retry after uncertainty. Work may be lost, but duplicates are avoided. Appropriate only when loss is cheaper than duplication or can be repaired elsewhere.
• At-least-once: retry until acknowledged. Duplicates are expected, so handlers must make repeated delivery harmless.
• Effectively once: use idempotency, uniqueness, and atomic state transitions so the observable business result occurs once within a defined boundary.

Kafka supports idempotent production and transactions within its own processing model, but an interview answer should not stretch that guarantee across an arbitrary email, payroll API, and database. Define the transaction boundary and compensation path. The official Kafka delivery-semantics documentation is a useful vocabulary check.

Inbox, outbox, and idempotent effects

BEGIN;
INSERT INTO processed_event(tenant_id, event_id)
VALUES (:tenant, :event)
ON CONFLICT DO NOTHING;             -- duplicate becomes a no-op

-- Continue only if one row was inserted.
UPSERT employee ...;
INSERT INTO outbox(event_id, aggregate_id, event_type, payload) ...;
COMMIT;

The inbox prevents repeated consumption from applying the same state transition. The outbox stores a domain event in the same database transaction as the domain change; a relay publishes committed rows and records attempts. This removes the “database committed, publish crashed” dual-write gap. It does not eliminate duplicate publication, so consumers still deduplicate. AWS’s primary pattern guide makes both points explicit: transactional outbox pattern.

For non-idempotent external effects, use a provider idempotency key if available. Otherwise introduce a local operation record with a unique business key and a state machine, then reconcile uncertain outcomes by querying the provider before retrying. Never assume a client timeout means the remote call failed.

Ordering, backpressure, and dead letters

Global ordering is expensive and rarely required. Partition by the smallest aggregate that needs order—often tenant_id + employee_id—and include an aggregate version. A consumer can reject stale versions, buffer a short gap, or trigger a focused refresh. Backpressure is a correctness control: bound worker concurrency, honor provider rate-limit hints, apply exponential backoff with full jitter, and stop admitting optional backfills before real-time changes starve.

A dead-letter queue is quarantine, not a cemetery. Store error class, schema version, attempt history, first/last failure time, and a redacted payload reference. Provide replay tooling that revalidates authorization and schema, rate-limits release, and is itself audited. Permanent validation errors belong in a review flow; transient dependency failures normally remain on the retry path.

4. Worked system: a resilient HRIS-to-AI knowledge connector

This is a hypothetical system for interview practice, not a claim about Purnendu’s experience. A customer wants employee directory and policy documents synchronized from an HRIS into an access-controlled AI assistant. The assistant may be minutes stale, but must not expose one tenant’s or one employee group’s data to another.

Architecture and state

Key records are connection (tenant, provider, scopes, encrypted credential reference, token version, health), sync_job (cursor, high-water mark, state, counts), inbox_delivery, source_object (remote ID, source version, payload hash, tombstone), employee, outbox_event, and sync_anomaly. Every tenant-owned key begins with tenant_id; repository methods require tenant context rather than accepting an optional filter.

The incremental sync captures a start high-water mark, pages deterministically, and advances the committed checkpoint only after each page’s effects are durable. Webhooks reduce latency but are hints, not the sole truth. A periodic reconciliation compares provider IDs, counts by status, and sampled hashes; it discovers missed webhooks, silent permission loss, and drift. Deletions become tombstones and downstream removal events, with retention determined by policy.

Failure table

Python backend judgment

Use async def for libraries that expose genuinely asynchronous network or database operations, and offload CPU-heavy parsing or blocking SDKs rather than calling them on the event loop. FastAPI’s own guidance distinguishes waiting on I/O from CPU parallelism: concurrency and async/await. Bound concurrency with a semaphore, pass a deadline through calls, close clients in application lifespan hooks, and treat cancellation as part of control flow rather than a generic error.

async with asyncio.timeout(job.remaining_seconds()):
    async with provider_slots:             # protects provider and this process
        page = await client.list_people(cursor=job.cursor)
    await repository.apply_page_atomically(page, job_id=job.id)
    await repository.commit_checkpoint(job.id, page.next_cursor)

# Important: do not catch BaseException and swallow CancelledError.

Validate external payloads into strict adapter models, then map them to domain types. Unit-test pure mappings; integration-test database transactions and queue redelivery; contract-test representative provider fixtures; and run end-to-end tests against a fake provider that can inject 429s, timeouts, malformed pages, cursor loops, and duplicate webhooks. In a secondary stack, look for the same semantics: Node promises still need cancellation/timeouts and bounded concurrency; Go contexts still need propagation and prompt cancellation.

Syllabus checkpoint: complete connector and backend breadth

API surface and traffic controls

REST and GraphQL both need authentication, tenant scoping, pagination, stable error contracts, observability, and abuse protection. Rate limits should expose a documented budget and retry hint; clients should combine deadlines, exponential backoff, jitter, and circuit breakers without retrying authorization or validation failures. For API versioning, prefer additive evolution, explicit deprecation windows, contract tests, and telemetry proving old versions are no longer used before removal.

Queues are products with different delivery contracts

Kafka is a partitioned log with replay and consumer offsets; SQS is a managed queue with standard/FIFO choices and visibility timeouts; Google Pub/Sub uses acknowledged delivery and subscription-specific retention. Do not flatten them into “a message broker.” State ordering scope, duplication behavior, retention/replay, backpressure, dead-letter policy, maximum payload, and the idempotent side-effect boundary. An RBAC model should restrict who can publish, consume, replay, inspect payloads, and administer schemas per tenant or environment.

Python service depth

FastAPI dependency injection can provide verified principals, tenant context, database sessions, and policy objects, but keep side effects explicit and cleanup deterministic. Use background jobs only for best-effort work that may be lost with the process; durable business work belongs in a queue with status and recovery. Pytest fixtures should establish isolated state, while unit, integration, contract, duplicate-delivery, timeout, and cancellation tests cover different boundaries.

Secondary-stack fluency

For Node.js and TypeScript, understand the event loop, promises/cancellation conventions, type narrowing, runtime validation, and package risk. In Go, understand contexts, goroutines, channels, errors, interfaces, and resource ownership. React breadth should cover state, async server interaction, accessible loading/error states, and streaming UI safety. GraphQL needs resolver authorization and N+1 awareness. PHP matters only for a role that genuinely uses it, such as parts of the Automattic ecosystem. The goal is production-strong Python plus the ability to read, implement, and critically review one secondary stack—not to claim equal depth everywhere.

Interview playbook

Lead with the business invariant, then trace one request and one failure. A compact answer structure is BOUNDARY:

1. B — Business truth: source of truth, freshness, deletion, conflicts, and success metric.
2. O — Ownership and identity: tenant, principal, scopes, data classification, and audit actor.
3. U — Uncertainty: timeouts, duplicates, partial failure, ordering, and unknown outcomes.
4. N — Normalized contract: canonical model, API/job state, event envelope, and versioning.
5. D — Durability: inbox/outbox, checkpoints, idempotency boundary, and reconciliation.
6. A — Admission control: limits, backpressure, deadlines, retry budgets, and circuit breakers.
7. R — Recovery and rollout: replay, DLQ, credential rotation, canary tenants, rollback.
8. Y — Yardsticks: sync lag, completion rate, reconciliation drift, duplicates suppressed, and tenant-scoped errors.

Common traps are saying “exactly once” without a boundary, using email as an immutable identity, trusting a tenant ID from an unsigned payload, retrying every error, treating a DLQ as recovery, or drawing Kafka before clarifying scale. When coding, narrate cancellation, transaction scope, resource cleanup, and how the test proves duplicate safety.

Question bank

Practise these aloud. Each answer should state assumptions and defend one concrete boundary.

Proof artifact: resilient connector laboratory

Build a small, provider-neutral employee connector. All target numbers below are example acceptance thresholds, not claims of prior results.

1. Create a FastAPI control API for connection setup, sync-job creation, status, cancellation, and webhook intake. Use strict request/response models and a machine-readable error envelope.
2. Implement PostgreSQL tables for connection, job/checkpoint, inbox, canonical employee, outbox, and anomalies. Add tenant-scoped unique constraints.
3. Build a fake HRIS with cursor pagination, OAuth token expiry, configurable 429/500/timeout responses, mutable records, deletions, and signed duplicate webhooks.
4. Run a bounded worker that resumes from checkpoints, uses jittered retries and deadlines, atomically applies pages, and relays the outbox to a fake index.
5. Add reconciliation that reports source-only, destination-only, version, and payload-hash mismatches, then offers an audited targeted repair.

Measure: job completion and partial-failure rate, p50/p95 sync lag, queue age, provider calls per changed record, retry count by class, duplicate deliveries suppressed, reconciliation drift, DLQ age, and cross-tenant access test results. Example goals might be zero duplicate business effects in 1,000 repeated deliveries and checkpointed recovery without a full restart; label them as lab targets.

Inject failures: kill the worker after database commit but before acknowledgement; replay one webhook 100 times; expire a token while two workers refresh; return the same cursor twice; add an unknown enum; throttle one tenant; make indexing time out after applying the write; and disconnect a tenant during a backfill. Record expected, observed, and repaired state.

Present: a two-minute architecture walkthrough, state-machine and schema diagram, one trace across webhook-to-index, before/after reconciliation report, automated failure-test output, and a short decision record covering at-least-once delivery, tenant isolation, and one rejected alternative.

Chapter review

A production connector assumes change, duplication, delay, partial failure, and revoked authority. Its core is not the HTTP client: it is an explicit contract plus durable state, idempotent effects, bounded work, reconciliation, and auditable tenant-aware recovery.

Glossary

Canonical model

A stable internal representation that isolates the domain from provider-specific schemas.

High-water mark

A durable boundary showing how far an incremental process has safely advanced.

Idempotency

The property that repeating an operation with the same identity has no additional business effect.

Inbox / outbox

Durable tables that deduplicate received messages and atomically stage messages to publish.

Reconciliation

Comparison of source and destination truth to detect and repair drift that event delivery missed.

Retry budget

A bound on extra attempts so recovery traffic cannot overwhelm a degraded dependency.

Saga

A durable multi-step business process using local transactions and semantic compensation.

Tombstone

An explicit marker that a source object was deleted, enabling downstream removal and audit.

Mastery checklist

• I can define an idempotency key’s scope, retention, concurrency behavior, and stored result.
• I can trace tenant identity from authenticated ingress through queue, database, cache, and audit log.
• I can explain why outbox publication and end-to-end exactly-once effects are different claims.
• I can recover missed webhooks using checkpoints, overlap, replay, and reconciliation.
• I can classify errors into terminal, retryable, uncertain, and human-repair states.
• I can implement bounded async I/O and preserve cancellation and cleanup.
• I can demonstrate duplicate, outage, schema-change, and tenant-isolation tests.

1. PostgreSQL: begin with invariants and access paths

A senior data answer starts with what must remain true under concurrency. Tables, indexes, and transactions are mechanisms for those invariants—not independent checklist items.

Model stable facts, preserve uncertain input

Normalize entities that have independent identity and lifecycle: tenant, source, document, document version, processing run, chunk, and access grant. Use foreign keys and unique constraints for truths the database can enforce. Keep the immutable source object in object storage and a raw metadata reference or carefully bounded JSONB column for provider-specific fields. Promoting every uncertain field into a column creates migration churn; placing every stable relationship in JSON discards relational guarantees and makes query behavior harder to predict.

document(
  tenant_id, document_id, source_id, external_id,
  current_version_id, lifecycle_state, created_at, updated_at,
  UNIQUE (tenant_id, source_id, external_id)
)

document_version(
  tenant_id, version_id, document_id, content_hash,
  object_uri, parser_version, source_modified_at, status,
  UNIQUE (tenant_id, document_id, content_hash)
)

Put tenant_id in ownership and uniqueness keys, not only in a nullable filter. Row-level security can add defense in depth: when enabled, normal access must be allowed by a policy, as the current PostgreSQL row-security documentation explains. Still test connection-pool session state, roles that bypass RLS, background jobs, migrations, and administrative access. RLS is not a substitute for explicit tenant-aware application APIs.

Index for a concrete query

“Add an index” is not a diagnosis. Capture the representative query and parameters, table/index sizes, data distribution, concurrency, cache state, and latency percentiles. Use EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) in a safe environment: ANALYZE executes the query, including writes. Compare estimated versus actual rows at each node, loops, scan type, join algorithm, sorts/spills, heap fetches, and shared reads/hits. PostgreSQL’s official guide emphasizes that a plan is a tree and estimates depend on statistics: Using EXPLAIN.

Transactions, locks, and pools

PostgreSQL defaults to Read Committed, where each statement sees a snapshot at statement start. Repeatable Read provides a stable transaction view; Serializable detects executions that cannot be ordered safely and requires the application to retry serialization failures. The exact behavior is documented in Transaction Isolation. Choose by invariant, keep transactions short, update resources in a consistent order, inspect lock waits, and make retries idempotent. A deadlock victim is expected safety behavior, not evidence the database is broken.

Connection pools protect a finite database resource. Size them from database capacity across all replicas/workers, not from request count. Long model calls must not hold an open transaction or idle connection. Use a pool acquisition timeout, statement timeout, transaction timeout, and metrics for active, idle, waiting, and age. Pool exhaustion can look like a slow query even when no query has started.

2. Build ingestion as a replayable ledger

Messy enterprise data means malformed files, duplicate exports, password-protected PDFs, changing parsers, conflicting encodings, and tables whose visual layout carries meaning. A trustworthy pipeline never makes the only copy of the input its latest derived output.

Stages and contracts

1. Discover: enumerate source objects with a stable source ID, version/etag, modified time, and permission snapshot. Do not assume directory listing order is a checkpoint.
2. Land: stream bytes to immutable object storage while computing size and cryptographic content hash. Enforce file and decompression limits before expensive parsing.
3. Validate: detect actual media type, malware policy result, encryption, structural damage, and tenant/source authorization. Quarantine with a reason; do not silently skip.
4. Extract: select a versioned parser by media type. Preserve page, sheet, cell, bounding box, and OCR confidence so answers can cite the source.
5. Normalize: produce a canonical document representation. Keep extraction warnings and unknowns; do not fabricate clean text from missing data.
6. Publish: atomically point the document at a complete index generation only after validation. Old generation remains available for rollback.

ETL transforms before loading into the analytical destination and is useful when strict sanitization or a stable target schema is required. ELT lands raw data first and transforms in a capable data platform, improving reprocessing and exploratory flexibility. A robust document pipeline often combines them: immutable landing is ELT-like; security validation before broad availability is ETL-like.

Batch, stream, checkpoints, and deduplication

Streaming lowers freshness but increases state and operational complexity. Batch improves throughput and gives clean processing boundaries. Use events for fast discovery and a scheduled inventory for correctness. A checkpoint must name a durable completed boundary: source cursor plus tie-breaker, manifest ID, input partition, and pipeline version. Commit it only after all outputs for that boundary are durable.

Deduplicate at several levels: source object ID/version prevents repeat discovery, content hash detects identical bytes across exports, and deterministic derived IDs prevent duplicate chunks. Whether two tenants may share physical bytes is a security and encryption decision; logical documents and access controls remain separate. Do not use fuzzy text similarity as the only identity rule.

Lineage and data-quality gates

For every indexed chunk, answer: which tenant, source object, source version, byte hash, parser/OCR version, normalization version, page/region, chunker configuration, embedding model/version, processing run, and access-control snapshot produced it? That lineage makes a targeted replay possible when a parser defect affects only scanned PDFs.

Quality gates should distinguish hard failures from warnings: empty extraction, implausible page counts, unreadable percentage, duplicated pages, corrupted tables, missing mandatory columns, OCR confidence distribution, and access-control mismatch. Route ambiguous cases to human review with the original rendering and extracted representation side by side.

3. Containers and Kubernetes: make runtime intent explicit

Kubernetes can restart and route around failures only when the application exposes truthful health and resource behavior. Deployment YAML cannot compensate for an endpoint that lies.

Build a small, reproducible image

Use a multi-stage build so compilers and build caches do not enter the runtime image; Docker’s official guide shows how stages selectively copy artifacts: multi-stage builds. Pin base images by an intentional version or digest, run as a non-root user, use a read-only filesystem where possible, exclude secrets and build context, emit an SBOM/signature in the supply-chain workflow, and scan both dependencies and the final image. Rebuild images for patches; do not mutate running containers.

# Illustrative structure; pin reviewed versions/digests in a real build.
FROM python:3.13-slim AS build
WORKDIR /build
COPY pyproject.toml uv.lock ./
RUN ... build a locked wheelhouse ...

FROM python:3.13-slim AS runtime
RUN useradd --system --uid 10001 app
COPY --from=build /build/wheels /wheels
RUN ... install only locked runtime wheels ...
USER 10001
CMD ["python", "-m", "service"]

Three probes, three questions

• Startup: has this slow-starting process initialized enough for other probes to begin?
• Readiness: should this pod receive new traffic now? Overload or loss of a mandatory local capability may make it unready.
• Liveness: is the process irrecoverably stuck such that restart is likely to help?

Kubernetes suppresses readiness/liveness until a configured startup probe succeeds, and a failed readiness probe removes the pod from service endpoints. Its documentation also warns that bad liveness probes can cause cascading restarts under load: probe guidance. Do not make liveness depend on every remote model provider; restarting healthy pods during a provider outage increases damage.

Resources, scaling, and rollout

CPU requests influence scheduling and CPU limits can throttle; memory limits can end in OOM termination. Start from load-test profiles for API, parser, and worker separately. Observe working set, CPU throttling, garbage collection, queue age, and latency. Keep headroom for bursts and node disruption. A HorizontalPodAutoscaler adjusts replicas from observed metrics, but scaling on CPU alone can fail for I/O-bound queue workers; queue age or outstanding work per ready worker is often closer to user pain. See Kubernetes autoscaling concepts.

A rolling deployment needs enough surge capacity, readiness that reflects warm-up, termination grace, request/lease draining, and a disruption budget. Canary releases route a small, observable cohort to the new parser/model/service and compare errors, latency, cost, and quality before expansion. Blue-green gives a clean environment switch and fast rollback but doubles capacity during transition and does not magically reverse database migrations. Prefer expand/migrate/contract schemas and backward-compatible readers.

4. Cloud platform design: identity, network, delivery, and recovery

A reference architecture should explain traffic and trust, not display a cloud catalog. Trace both the data plane and the control plane.

One request from DNS to data

DNS resolves the service name; TLS authenticates the endpoint and encrypts transport; a load balancer or ingress applies routing and connection policy; the service authenticates the caller; workload identity authorizes narrowly scoped calls to database, object storage, queue, secret manager, and model provider. Network policy and private endpoints reduce paths but do not replace identity checks. Timeouts must descend: client deadline > ingress > service subcalls, leaving time to return a useful error.

Prefer short-lived workload identity to static cloud keys. Separate deployer identity from runtime identity. A worker reading source objects need not mutate infrastructure; an API serving job status need not decrypt every source credential. Centralize secrets in a managed store, rotate them, and record access without printing values.

Infrastructure as code and CI/CD

Terraform describes desired resources and records bindings in state. State can contain sensitive information and must be shared and locked safely; HashiCorp recommends remote state for team use and warns against insecure version-control storage: Terraform state documentation. Pin provider/module versions, review a saved plan, use separate state blast radii, run policy/security tests, and avoid broad production credentials in pull-request jobs.

1. Build once; produce immutable image digest, tests, SBOM, vulnerability result, and provenance.
2. Plan infrastructure and schema changes; require review for destructive or privilege-expanding operations.
3. Deploy to a representative pre-production environment; run smoke, contract, migration, and rollback tests.
4. Canary by environment, tenant cohort, or traffic; automatically pause on user-facing SLO and correctness signals.
5. Promote the same artifact, then verify and retain evidence. Rollback or roll forward through an exercised procedure.

Public cloud favors managed-service velocity and elastic capacity; private or customer-hosted deployments may be required for data residency, network control, or procurement policy, but increase version skew, upgrade, capacity, and support burdens. Hybrid design needs an explicit connectivity failure mode and a support boundary.

Backups are not recovery

Define recovery point objective (maximum acceptable data loss) and recovery time objective (maximum acceptable restoration time) by component. Test database point-in-time recovery, object versioning, queue redrive, index rebuild from canonical data, infrastructure recreation, identity/secrets restoration, and DNS failover. Record actual drill times. A replica can copy corruption; a backup can be unusable; an index may be cheaper and safer to rebuild than back up.

5. Worked system: multi-tenant document processing platform

This hypothetical example turns the prior decisions into a system-design narrative. The numbers are illustrative sizing assumptions, not production results.

Estimate before selecting capacity

Assume 100 tenants, 50,000 documents each, four pages per document: 20 million pages initially. If 0.5% change daily, steady state is about 100,000 pages/day, but an onboarding backfill may be 50 times the average. If OCR consumes an example 1.5 CPU-seconds/page, the daily steady-state CPU work is about 42 CPU-hours; a 10-hour processing objective needs roughly 4.2 continuously busy cores before concurrency inefficiency, retries, and headroom. Benchmark the real corpus before committing.

Store original bytes and manifests in object storage, canonical metadata/checkpoints in PostgreSQL, and tasks in a durable queue. Separate fetch, virus/format validation, OCR/extraction, normalization/chunking, embedding, and index-publish workers so each scales and retries independently. Use deterministic artifact keys and an index generation pointer. Interactive query traffic runs in a different deployment and resource pool from ingestion.

Capacity and cost model

Protect fairness with per-tenant quotas and weighted scheduling. Large backfills consume a separate budget and can pause when interactive latency or database saturation rises. Operational dashboards join queue age, stage throughput, success/warning/quarantine rates, CPU/memory, database waits, provider usage, cost per usable page, and freshness by tenant.

Deploy in one region first if requirements allow, using multi-zone managed services. Keep raw and canonical data sufficient to rebuild derived indexes. For regional disaster recovery, choose active-passive unless the recovery objective justifies active-active data consistency and operational complexity. Explain how tenant data residency changes placement and how the control plane routes a tenant to the correct deployment stamp.

Syllabus checkpoint: database, Kubernetes, Linux, and cloud breadth

PostgreSQL beyond one slow query

Schema design starts from invariants, update patterns, and query grain; normalization reduces contradictory facts, while deliberate denormalization needs an ownership and refresh rule. PostgreSQL full-text search uses document/query representations, dictionaries, ranking, and indexes and can complement pgvector for hybrid retrieval. Test migrations forward and backward against production-like data, and test backups by restoring them to a clean environment.

Configuration and delivery on Kubernetes

ConfigMaps hold non-secret configuration; Secrets are transport/storage objects whose encryption, RBAC, rotation, and workload delivery still need design. Prefer workload identity to long-lived cloud keys. Helm packages parameterized Kubernetes resources; GitOps reconciles declared state through reviewed changes. Neither replaces readiness tests, safe database migration order, canary or blue-green analysis, termination/draining, nor a verified rollback.

Linux and networking diagnosis

Trace a request through DNS resolution, TCP connection, TLS handshake, proxy/load balancer, service routing, application, and downstream dependency. On Linux, inspect process state, sockets, CPU, memory, disk and inode pressure, file descriptors, cgroups, logs, and permissions before changing configuration. Distinguish connection timeout, refusal, reset, TLS verification, and application timeout; they imply different layers and owners.

AWS and GCP mapping

Map requirements to durable primitives before provider names: identity/IAM, network boundary, compute, object storage, queue/event service, managed PostgreSQL, Kubernetes/serverless, secrets/KMS, monitoring, and audit. AWS and GCP differ in service mechanics and defaults, so validate the chosen managed service’s quotas, availability model, backup/restore, private networking, egress, and pricing. Terraform should produce reviewed, repeatable state with remote locking, least-privilege credentials, drift detection, and a recovery plan for state—not merely create resources.

Interview playbook

Use QUERY → PIPELINE → PLATFORM:

1. Query: identify invariants, access patterns, scale, distribution, and transaction boundary; sketch keys before indexes.
2. Pipeline: show immutable input, versioned stages, idempotent IDs, checkpoint, quarantine, lineage, and replay.
3. Platform: trace network and identity, stateful dependencies, probes/resources, scaling metric, rollout/rollback, recovery, and cost.

For a slow-query question, ask for plan and data distribution before proposing an index. For Kubernetes, do not confuse liveness with readiness or autoscaling with capacity. For cloud architecture, name RPO/RTO and trust boundaries. State illustrative numbers as assumptions, show the arithmetic, and explain what benchmark would replace them.

Common traps include using JSONB for every field, running EXPLAIN ANALYZE on a dangerous production write, holding a database transaction across OCR/model calls, committing a checkpoint before outputs, making liveness depend on a remote provider, setting resources by guesswork, scaling workers until the database fails, and claiming a backup without a restore test.

Question bank

Answer with one concrete workload and measurable verification.

Proof artifact: operable document platform

Build a local or low-cost reference deployment. Any thresholds are example lab objectives, not claims about past work.

1. Generate a synthetic tenant-safe corpus containing clean text PDFs, scans, malformed PDFs, CSV/Excel edge cases, duplicates, deletes, and deliberate cross-tenant IDs.
2. Implement immutable landing, manifest/checkpoint tables, versioned extraction, quarantine, deterministic chunk IDs, lineage, and a mock index-generation switch.
3. Create one deliberately slow PostgreSQL workload. Save schema, data generator, query, EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON), change hypothesis, new plan, and load-test comparison.
4. Containerize stages with a multi-stage non-root image. Deploy API and workers to a local Kubernetes cluster with startup/readiness/liveness, requests/limits, HPA or event-based scaling, graceful shutdown, and a rollback command.
5. Describe infrastructure in Terraform for a disposable environment or use a safe mock plan; keep state outside version control and document identity boundaries.

Measure: usable pages per minute, p95 stage latency, queue oldest age, quarantine/warning rate, checkpoint recovery time, duplicate suppression, database plan rows/buffers/time, CPU throttling, memory peak/OOM, rollout error rate, estimated cost per 1,000 usable pages, and restore/rebuild time.

Inject failures: kill a worker between artifact write and checkpoint, corrupt a file, force OCR timeout, exhaust the database pool, introduce one tenant with a huge backfill, fail readiness, OOM a parser under a limit, interrupt a rollout, and rebuild the index from canonical state. Prove that a poison document does not block its partition and a tenant cannot read another tenant’s manifest.

Present: a scale worksheet, data/lineage diagram, before/after plan with reasoning, Kubernetes manifest excerpt, one failure timeline, recovery evidence, cost sensitivity chart, and an architecture decision record for batch versus streaming or shared versus deployment-stamp isolation.

Chapter review

An operable AI data platform preserves raw truth, makes transformations versioned and replayable, enforces database invariants, exposes honest runtime health, and treats delivery, recovery, and cost as design inputs.

Glossary

Execution plan

The planner’s tree of scan, join, sort, and aggregation operations, with estimated or measured work.

Isolation level

The visibility and anomaly guarantees a transaction receives under concurrency.

Lineage

The trace from a derived artifact back through versions, transformations, and source input.

Manifest

A durable inventory of inputs and outputs for one processing boundary or generation.

Quarantine

An isolated state for unsafe or invalid input that preserves evidence and prevents publication.

Readiness

Whether a workload should receive new traffic now; distinct from whether its process should restart.

RPO / RTO

Maximum acceptable data loss and maximum acceptable restoration time.

Workload identity

A short-lived identity assigned to running software for authorized service access.

Mastery checklist

• I can derive schema keys and indexes from invariants and access patterns.
• I can interpret estimated versus actual rows, loops, buffers, waits, and pool pressure.
• I can resume an ingestion run without duplicate publication and trace every chunk to source.
• I can explain probe controller actions and demonstrate graceful shutdown.
• I can choose a scaling signal while protecting downstream capacity and tenant fairness.
• I can separate application, schema, artifact-generation, and infrastructure rollback.
• I can state RPO/RTO and show a tested restore or rebuild path.
• I can produce a cost range with assumptions and sensitivity rather than a false-precision total.

1. Reliability is a user-visible contract

“The pods were up” is not a product outcome. A user needs an authorized, sufficiently correct answer or a truthful, recoverable failure within an acceptable time. Define reliability at that boundary.

SLI, SLO, SLA, and error budget

• A service-level indicator (SLI) is a measured ratio or distribution, such as valid successful assistant requests divided by eligible requests.
• A service-level objective (SLO) is a target over a window, such as an illustrative 99.5% of eligible requests producing a valid response within a specified latency threshold over 28 days.
• A service-level agreement (SLA) is a business/legal commitment and may use different definitions or consequences.
• An error budget is the allowed unreliability: for a 99.5% example SLO, 0.5% of eligible events. It becomes useful only when a written policy changes rollout and engineering decisions.

Google’s SRE guidance emphasizes user-centered targets and organizational backing for error-budget consequences: The Art of SLOs. Do not set an SLO from aspiration alone. Examine user tolerance, dependency capability, cost, historical performance, and what action the team will take when the budget burns.

Define eligibility and “good” precisely

Use request-based SLIs for interactive traffic and window/backlog-age SLIs for pipelines. For latency, a histogram or event distribution preserves tail behavior; an average can stay healthy while one customer cohort suffers. Define the measurement point and denominator so client disconnects, policy denials, and dependency timeouts cannot be reclassified opportunistically.

Burn rate turns a monthly target into an alert

Burn rate is observed bad-event rate divided by allowed bad-event rate. A service consuming one day’s budget each day burns at 1×. Multi-window alerts combine a short window that detects fast incidents with a longer window that filters transient noise. Page on budget-threatening user impact; ticket on slower trends; dashboard everything else. Exact thresholds depend on the SLO and response model.

2. Observability: connect symptoms to causes

Monitoring asks known questions; observability lets an engineer investigate unanticipated states from system outputs. Instrument a coherent event model, not three disconnected vendors.

Logs, metrics, traces, and context

• Metrics aggregate rates, errors, durations, saturation, queue age, token/cost use, and quality samples cheaply enough for dashboards and alerts.
• Traces show causality and time across ingress, retrieval, model calls, tools, databases, queues, and policy checks.
• Logs explain discrete state transitions and diagnostics with structured fields.
• Baggage/context carries selected correlation metadata across process boundaries; it must be size-bounded and must not carry secrets or raw PII.

OpenTelemetry currently defines traces, metrics, logs, and baggage as supported signals: OpenTelemetry signals. Use one resource identity (service.name, version, environment, region), propagate W3C trace context through HTTP and message metadata, and retain application-level IDs such as job or conversation ID in a privacy-safe form.

Trace an AI request without logging the world

Record duration, status/error class, model/provider route, token counts, cache outcome, retrieval counts/scores, tool name and outcome, approval decision, validation result, and policy version. Default to hashes, classifications, counts, and references rather than prompts, retrieved text, model output, secrets, or tool arguments. Provide an explicitly authorized, short-retention diagnostic mode for rare cases, with access logs and redaction.

For queued work, the producer span ends before the consumer starts; propagate context in the message and consider a span link when work is batched or fan-outs merge. Trace the retry attempt separately while keeping a shared logical operation ID. Otherwise a three-attempt dependency call looks like one slow span and hides amplification.

Cardinality, sampling, and useful dashboards

Metric label values must remain bounded. model_route or normalized error_class may be useful; user_id, prompt text, document ID, URL, or exception message can create unbounded series and cost. Keep high-cardinality correlation in traces/logs under access control. Use histograms for latency rather than client-side percentile labels; review the official Prometheus histogram guidance for aggregation trade-offs.

Head sampling decides before a trace finishes and is cheap but may miss rare failures. Tail sampling can retain errors, high latency, or important cohorts after observing the trace but requires buffering and collector capacity. OpenTelemetry documents these choices in Sampling. Preserve enough unbiased baseline traffic to estimate rates; an errors-only trace store cannot reveal how unusual an error path is.

A service dashboard should follow user journey → dependencies → resources: SLO and burn, traffic, error classes, latency, quality/freshness, queue age, model/tool outcomes, saturation, and deployment annotations. An alert must say what user promise is threatened, the affected scope, likely first checks, and runbook; if no one should act now, it is not a page.

3. Resilience is a coordinated control loop

Timeouts, retries, breakers, queues, and fallbacks interact. Configure them as a budgeted system; independent defaults often turn one slow provider into a fleet-wide outage.

Deadlines first, then retry

A deadline is the total time the caller is willing to wait. Each downstream timeout must fit inside the remaining deadline and leave time for cleanup or a useful response. Connect, request, stream-idle, and pool-acquisition timeouts protect different waits. Propagate cancellation so abandoned work does not continue spending tokens and database capacity.

Retry only errors likely to improve on another attempt, only when the operation is safe or idempotent, with exponential backoff and full jitter, a capped attempt count, and a shared retry budget. Honor provider rate-limit guidance. If three service layers each retry three times, the lowest dependency may see up to 27 attempts for one user request; choose one owner for retries or coordinate them.

Protection patterns and their limits

Graceful degradation should preserve truth. Examples: answer from a verified cache with a visible age; switch from an agentic write flow to read-only retrieval; queue a document update and show delayed status; return a cited search result instead of generating; or fail closed for a high-risk tool. A smaller model is not automatically safe: re-run policy and quality gates, disclose capability differences where relevant, and ensure it supports the required region and data terms.

Capacity, disaster recovery, and dependency isolation

Load testing finds throughput and latency under expected mix; stress testing finds the failure boundary; soak testing exposes leaks and slow degradation. Include model latency distributions, streaming connections, long documents, retries, tenant bursts, and cold caches. Capacity plans reserve headroom for failover: if one zone fails, remaining capacity must carry critical load without triggering autoscaling too late.

Set RPO/RTO per state. Conversation records may need point-in-time database recovery; derived vectors may be rebuilt; queued tool actions may require reconciliation before replay; provider credentials may need separate secured recovery. Exercise failover and restore, including the route back to primary. “Multi-region” without conflict, identity, secret, data-residency, and failback design is a diagram, not a recovery plan.

4. AI security: constrain authority, not just text

An LLM processes instructions and untrusted data in the same medium. Prompt rules are useful behavior guidance, but authorization must be enforced by deterministic systems outside the model.

Threat-model the complete flow

List assets (tenant documents, credentials, tool authority, model inputs/outputs, audit evidence), actors (user, tenant admin, insider, compromised document/source, provider, operator), trust boundaries, entry points, and abuse outcomes. Then trace data and authority through retrieval, memory, prompt construction, model, tool broker, downstream API, and logs.

The OWASP 2025 LLM guidance identifies prompt injection and excessive agency as distinct but related risks. Its excessive-agency mitigations emphasize minimizing tool functionality, permissions, and autonomy and enforcing downstream authorization: OWASP Excessive Agency. Do not claim input filtering “solves” prompt injection; reduce the impact when the model is influenced.

Tool broker pattern

The model proposes; the broker decides. Bind each action to authenticated tenant/user, purpose, resource scope, idempotency key, monetary or row limit, and expiry. Resolve opaque resource IDs server-side instead of letting the model supply arbitrary URLs or SQL. High-impact actions receive a human-readable preview based on validated parameters, not free-form model prose. Approval must bind to the exact action digest so parameters cannot change afterward.

Enterprise privacy and evidence

Classify data before choosing model/provider and region. Document data-processing purpose, storage/retention, training/use terms, subprocessors, encryption, residency, deletion, incident handling, and access. Minimize prompts, redact when compatible with the task, and separate customer content from operational telemetry. Rotate secrets and encryption keys through an exercised procedure; never make raw secrets model context.

Audit records should answer who or what principal acted, for which tenant, under which policy/model/tool versions, on which resource, with what approval, outcome, and correlation ID. Avoid storing the sensitive payload when a hash/reference and separately controlled evidence store suffice. Retention and legal requirements vary by customer and jurisdiction; do not turn awareness of GDPR, SOC 2, or ISO 27001 into a claim of compliance. NIST’s Generative AI Profile organizes voluntary risk work across govern, map, measure, and manage: NIST AI 600-1.

5. Worked incident: provider latency becomes a retry storm

This is a hypothetical interview scenario. Example times and measurements illustrate how to tell an incident story; they are not Purnendu’s experience or results.

Detection and mitigation

At 10:02 UTC, fast-burn alerts fire for interactive-answer latency and valid-success SLOs. Queue age and model-call attempts rise, while application CPU remains moderate and provider first-attempt latency rises. A deployment marker shows no internal release. Traces reveal that gateway, orchestration service, and SDK each retry the same timeout, amplifying attempts.

1. Declare incident, assign incident commander, operations lead, communications lead, and scribe; preserve a shared timeline.
2. Disable lower-level retries through dynamic configuration, reduce per-request attempt budget, open a route-scoped circuit, and shed noncritical batch traffic.
3. Route eligible low-risk requests to a previously evaluated fallback; fail closed for unsupported tools and expose a clear retryable status.
4. Protect recovery by limiting client retry guidance, monitoring fallback capacity/quality, and keeping one controlled probe path to the primary.

By the illustrative 10:18, burn rate falls; by 10:40, the primary is stable but traffic is restored in steps. The team verifies SLO, queue drain, error mix, fallback quality, and cost before resolving. Customer communication states observed impact and current mitigation without speculating about root cause.

Root cause versus trigger

The provider slowdown is the trigger. The internal root cause of severity is uncoordinated retries without a propagated deadline or shared budget, plus a fallback path whose capacity alarm was missing. Contributing conditions include a timeout longer than the upstream request budget, an alert on CPU rather than attempt amplification, and a runbook that did not identify retry owners.

Corrective actions should have owners and verification: one retry layer; deadline propagation test; attempt-count metric; dependency-specific bulkhead; fallback load/quality drill; client retry contract; and a chaos scenario in release qualification. Avoid “be more careful.” A blameless postmortem holds the system and decisions accountable while creating conditions for truthful reporting.

Syllabus checkpoint: operations and enterprise identity

From structured logs to an on-call decision

Structured logs capture discrete, queryable events with stable fields; metrics summarize rates and distributions; traces connect causal work across retrieval, model, tool, queue, and database boundaries. OpenTelemetry provides shared context and export, Prometheus stores/scrapes metrics, and Grafana commonly visualizes and alerts across data sources. Tool choice is secondary to cardinality control, redaction, sampling, retention, and a trace ID that connects the user-visible failure to evidence.

An on-call system needs severity definitions, ownership, escalation, runbooks, safe mitigations, communication cadence, and post-incident follow-through. Alerts should describe a user symptom and an action, not every internal anomaly. Test alert delivery and runbooks during failure drills; an unexercised pager path is not a control.

OAuth/OIDC, IAM/RBAC, and key rotation

OAuth delegates access; OIDC adds an identity layer and ID-token semantics. IAM defines principals and permissions across the platform, while application RBAC maps verified identity to domain roles—often with attribute checks for tenant, resource, or risk. Keep authorization server-side and test denied paths. Key rotation needs overlapping validity, versioned key identifiers, atomic rollout, detection of stale consumers, revocation for compromise, and an audit trail; “replace the secret” is not an operational plan.

Interview playbook

Use the PROMISE framework:

1. P — Promise: user journey, SLI, SLO, eligibility, window, and policy.
2. R — Risks: dependency, overload, data, security, privacy, and operator failure modes.
3. O — Observability: correlation, metrics/logs/traces, sampling, redaction, dashboards, alerts.
4. M — Mitigation: deadlines, isolation, backpressure, load shedding, safe fallback, and fail-closed cases.
5. I — Incident: roles, timeline, evidence, communication, and recovery verification.
6. S — Security: assets, trust boundaries, least authority, deterministic mediation, audit, and retention.
7. E — Exercise: load/chaos/security tests, restore drills, measured outcomes, and corrective ownership.

Common traps are defining availability at the pod, paging on every error, using raw user IDs as metric labels, storing full prompts by default, stacking retries, treating fallback as merely a cheaper model, claiming prompt injection is prevented by a system prompt, trusting the model to authorize its own tools, listing compliance acronyms as controls, or ending an incident at mitigation without root cause and verification.

Question bank

Practise answers that join reliability, observability, and security rather than treating them as separate teams.

Proof artifact: production ownership drill

Instrument a small RAG-plus-tool service or the chapter 06 platform. Use synthetic data. Any numeric thresholds are example lab objectives.

1. Define two user journeys with SLI equations, eligibility, example SLOs, error-budget policy, fast/slow burn alerts, and runbooks.
2. Add OpenTelemetry traces across API, queue, retrieval, model, policy, approval, and tool; export metrics and structured redacted logs. Document attribute allowlists and retention.
3. Implement deadlines, one retry owner, retry budget, bounded queues, per-tenant bulkhead/quota, circuit breaker, and one evaluated degraded mode.
4. Create a data-flow threat model and abuse cases for indirect injection, excessive agency, cross-tenant retrieval, secret leakage, and malicious tool output. Add typed tool mediation and an approval digest.
5. Write an incident template with roles, timeline, impact updates, mitigation decision log, root-cause tree, and corrective-action verification.

Measure: SLI and burn rate, attempts per logical request, p50/p95/p99 latency, queue oldest age, circuit state, shed/degraded requests, fallback quality/cost, trace/log drop rate, active metric series, policy denials, cross-tenant test failures, and recovery time. Keep labels bounded.

Inject failures: disable the model provider; add 2-second latency; return 429s; fill a queue with one tenant; kill the telemetry collector; corrupt a retrieved document with tool instructions; try arbitrary tool parameters; expire approval; leak a fake secret and verify redaction; restore state; and canary recovery. Capture what the user sees and whether the error budget stops release.

Present: SLO sheet, dashboard and alert screenshot, one end-to-end trace, redaction/cardinality tests, threat-model diagram, tool-policy test, incident timeline, root-cause tree, corrective action with owner/test, and a three-minute live dependency-failure drill.

Chapter review

Production ownership connects a measurable promise to telemetry, bounded failure controls, deterministic authority, practiced response, and verified recovery. Reliability and security both reduce uncontrolled blast radius.

Glossary

Burn rate

Observed bad-event rate divided by the rate allowed by an SLO.

Bulkhead

Resource isolation that prevents one workload, dependency, or tenant from exhausting all capacity.

Deadline

The total remaining time within which an operation remains useful to its caller.

Excessive agency

Risk created by giving an AI system more functionality, permission, or autonomy than its task requires.

High cardinality

A label/attribute dimension with many or unbounded distinct values.

Prompt injection

Untrusted content influencing model behavior contrary to the application’s intended instruction hierarchy.

SLI / SLO

A measured service indicator and its target over a defined population and time window.

Tail sampling

Selecting traces after enough of their outcome is known to apply error or latency policies.

Mastery checklist

• I can write an SLI equation with an honest denominator and cohort slices.
• I can connect error-budget burn to a release and incident policy.
• I can trace an AI/tool request without retaining unnecessary sensitive content.
• I can bound labels, sampling, retries, queues, concurrency, and fallback capacity.
• I can explain which actions degrade, queue, reject, or fail closed.
• I can enforce tool authorization outside the model and bind human approval to an exact action.
• I can distinguish an incident trigger, root cause, contributors, and verified corrections.
• I have exercised restore, dependency failure, cross-tenant access, and prompt-injection scenarios.

1. Discovery before diagrams

A Forward Deployed Engineer operates where customer process, messy data, security policy, product capability, and production engineering meet. The first deliverable is a shared problem definition. Drawing components too early hardens guesses into architecture.

Move from request to decision

A customer may ask, “Build an AI assistant for all our policies.” Clarify the job: Who asks which questions? What decision or task follows an answer? Which source is authoritative? Is a citation mandatory? How stale may content be? What must the system refuse? Is the assistant read-only, or may it take actions? What existing workflow, cost, or risk should change?

Write assumptions as tests

Maintain an assumption log with owner, evidence, risk if wrong, and validation date. “Documents contain reliable ACL metadata” becomes: sample 500 representative objects across systems; compare source entitlements with retrieved results; require zero unauthorized returns before pilot. “The model is accurate” becomes an evaluation dataset segmented by task and risk, an acceptance threshold, and human adjudication for disagreement.

Define done at four levels:

1. Functional: supported journeys and explicit non-goals.
2. Quality and safety: groundedness/task success, policy compliance, access control, and human escalation.
3. Operational: latency, availability/freshness, recovery, observability, runbook, support owner.
4. Value/adoption: eligible users, sustained usage, process outcome, and a measurement design that avoids misleading attribution.

2. A repeatable system-design method

Strong system design is structured uncertainty reduction. Use the same sequence in a 45-minute interview and a customer architecture workshop, changing only depth.

Requirements, estimates, contracts, flows

1. Frame: users, core use cases, non-goals, system boundary, source of truth, and success.
2. Quantify: tenants/users, requests per second, concurrency, objects/bytes/vectors, churn, payload size, fan-out, growth, peak factor, latency, SLO, RPO/RTO, and budget range.
3. Define contracts: external APIs/events, job state machines, core records, identity and authorization context, idempotency and versioning.
4. Trace flows: one normal read/write and the highest-risk asynchronous flow, including checkpoints and audit.
5. Choose architecture: components only after their responsibility is clear; identify data/control planes and trust boundaries.
6. Stress: overload, dependency loss, duplicate/out-of-order events, schema/model migration, tenant leak, regional failure, human/operator error.
7. Operate and evolve: SLIs, capacity/cost, deployment, canary, rollback, migration, ownership, and rejected alternatives.

Estimate with ranges, not theatre

Suppose an illustrative design has 50,000 users, 10% active in a peak hour, and six assistant turns per active user: about 30,000 requests/hour or 8.3 requests/second average during that hour. Apply an example 3× burst factor: roughly 25 requests/second. If each request retrieves 20 candidates and sends an average 8,000 total tokens through the model, provider throughput and cost—not API CPU—may dominate. Show the arithmetic, label every assumption, and state which load test or billing sample will replace it.

Capacity is multi-dimensional: requests/second, simultaneous streams, tokens/minute, queue age, database connections, index working set, OCR CPU, provider quotas, and human approval throughput. A system can have idle CPU while blocked on tokens/minute or a saturated connection pool.

Make interfaces carry correctness

POST /v1/assistant/turns
Authorization: Bearer ...
Idempotency-Key: ...
{
  "conversation_id": "...",
  "message": "...",
  "requested_tools": ["policy_lookup"],
  "client_context": {"locale": "en-IN"}
}

The server derives tenant, user, roles, policy and quotas from auth.
The client never supplies a trusted tenant_id or unrestricted tool URL.

Separate synchronous admission from long-running work. Version events and prompts/policies. Give every action and artifact a stable ID, lifecycle state, actor, tenant, provenance, and timestamps. Prefer deterministic downstream authorization over asking the model whether access is allowed.

3. Eight practice architectures and their hard parts

The syllabus’s systems share primitives, but each has a different correctness center. In an interview, spend time where failure is uniquely expensive.

Isolation is a spectrum

A fully shared deployment is cost-efficient and operationally simple but relies heavily on correct logical isolation and fairness. Per-tenant infrastructure improves blast-radius and configuration isolation but costs more and creates fleet-management/version-skew work. Deployment stamps group selected tenants and provide a scalable middle ground. Microsoft’s current multitenancy guidance frames the choice as trade-offs among isolation, cost, scale, performance, complexity, and manageability: architectural approaches for multitenancy.

Decide separately for compute, database, object storage, vector index, queues, encryption keys, network, and model route. A regulated tenant might have a dedicated data plane while sharing a global control plane. The control plane provisions connections, policies, tenants, and deployments; the data plane serves tenant traffic. Compromise or overload of one should not grant authority over the other.

Migrations are systems, too

For a vector-index migration, snapshot a source boundary, bulk backfill into a versioned target, capture concurrent changes via an ordered log/outbox, validate counts and sampled recall/ACL results, shadow or dual-read, canary tenants, then switch a routing pointer. Retain rollback until the old index’s update stream and retention window can safely close. Dual-write alone is not proof: one side can fail silently, so reconciliation is mandatory.

For API, schema, prompt, model, or tool migrations, state compatibility direction and rollback unit. Backward-compatible expand/migrate/contract often beats a flag day. Record the migration version with outputs so evaluation and audit can reproduce behavior.

4. Worked design: a multi-tenant policy assistant with approved actions

This hypothetical system demonstrates a complete interview answer. It is not a claim about Purnendu’s delivered projects or metrics.

Requirements and illustrative scale

Employees ask policy questions with citations and may propose a small set of HR service actions. Answers must honor source permissions and regional retention. High-impact writes require human confirmation. Assume for sizing 200 enterprise tenants, 100,000 total users, 40 peak assistant requests/second, 2 million source documents, a 15-minute freshness target for changed policies, 99.9% illustrative serving availability, and region-specific data planes. Confirm all values in discovery.

Non-goals for the first release: open-ended web browsing, arbitrary SQL or HTTP tools, autonomous approval, payroll decisions, and training on customer content. Success combines evaluated grounded-answer quality by policy domain, zero unauthorized retrieval in adversarial testing, latency/SLO, pilot adoption, and an agreed workflow outcome measured against a baseline.

Architecture

The global control plane maps a tenant to a deployment stamp and manages versioned configuration, but carries no customer prompt content. The regional gateway validates identity, derives tenant/user/roles, applies quotas, and routes only to the mapped data plane. The assistant orchestrator stores a checkpointed turn state with prompt/policy/model/retrieval versions and deadline.

Read and action flows

1. Normalize the user request and run policy/risk classification. Derive authorization filters from identity.
2. Run hybrid lexical/vector retrieval, pre-filter by tenant and ACL where supported, rerank, then reauthorize source fetch. Include provenance and current source version.
3. The LLM gateway selects an allowed model route by capability, region, policy, health, and budget. It enforces deadline/token limits and records usage without raw content by default.
4. Validate the answer structure, citations, policy, and uncertainty. If evidence is insufficient, return a scoped refusal or escalation rather than inventing.
5. For an action, the model emits a typed proposal only. The tool broker reauthorizes the user, validates allowlisted resource IDs and limits, creates an idempotent pending action, and renders a deterministic approval preview.
6. Approval is bound to action digest, approver, tenant, and expiry. Execution uses user-context or narrowly scoped service credentials, then records a receipt. Uncertain remote results enter reconciliation before retry.

Failure, security, and operations

Cost controls include per-tenant/model token budgets, prompt and retrieval limits, cache only where identity/policy/version keys make reuse safe, small-model routing for evaluated classes, incremental ingestion by content hash, and storage lifecycle. Report cost per successful eligible task alongside quality; optimizing cost per raw request rewards cheap failures.

Rejected alternatives

• One model call with all tenant documents: rejected for context limits, cost, stale data, poor provenance, and access-control risk.
• One dedicated stack per tenant from day one: rejected as the universal default because fleet cost and upgrades grow quickly; retained for isolation/residency tiers.
• Let the model call the HRIS directly: rejected because free-form authority, secrets, approval, idempotency, and audit cannot be enforced reliably.
• Active-active global writes immediately: rejected unless recovery requirements justify conflict, replication, residency, and operational complexity.

5. Forward Deployed execution: from pilot to durable ownership

FDE work succeeds when the customer can operate, trust, and extend the system after the initial team leaves. Technical depth and change management are one delivery problem.

Engagement sequence

1. Align: stakeholder map, sponsor, user owner, security/data/platform owners, decision process, RACI, outcomes, constraints, and working cadence.
2. Discover: workflow observation, representative data/ACL sample, architecture/security review, baseline measurement, and assumption/risk register.
3. De-risk: thin vertical slice against the hardest unknown—often access-correct retrieval, malformed documents, tool authorization, or deployment connectivity.
4. Pilot: limited users/data, shadow or read-only mode, explicit acceptance criteria, support channel, daily evidence review, and kill switch.
5. Productionize: SLOs, capacity, threat model, incident/restore drills, runbooks, observability, cost guardrails, and ownership training.
6. Roll out: cohorts with canary metrics, change communication, user education, feedback triage, rollback gates, and decision log.
7. Handoff and expand: architecture record, operations pack, backlog, known limitations, support escalation, outcome review, and next hypothesis.

Migration and workshop artifacts

A useful architecture workshop produces a context/data-flow diagram, requirements and NFR table, identity/ACL map, scale worksheet, risk/assumption register, option matrix, decisions and rejected alternatives, rollout plan, and owners. A migration assessment adds inventory, data quality, dependencies, compatibility, cutover/rollback, parallel-run duration, validation/reconciliation, downtime, training, and decommission criteria.

Do not make the proof of concept a hidden production system. Mark synthetic versus customer data, temporary credentials, retention, unsupported scale, missing controls, and expiry. Promotion requires an explicit review against production criteria rather than enthusiasm.

Handle objections and unsafe requirements

Listen for the underlying need, restate it, and separate non-negotiable safety from negotiable implementation. “No human approval because it slows the workflow” may conceal a latency goal. Offer risk-tiered automation: auto-execute bounded reversible low-risk actions; batch approvals; improve preview UX; keep high-impact irreversible actions gated. Quantify residual risk and seek the authorized risk owner’s decision. Never quietly implement an unsafe exception.

When a request conflicts with product capability or evidence, say: what is known, what is unknown, the failure/blast radius, the recommended safe path, alternatives, and the decision required. Escalation is a delivery skill when it preserves trust and schedule.

Measure value without overselling causality

Choose one primary outcome close to the workflow—such as eligible cases resolved with verified citations or time from accepted request to confirmed completion—and guardrails for quality, safety, cost, and equity across cohorts. Establish baseline and instrument eligibility before rollout. Use phased cohorts or a credible comparison when possible; report adoption and outcome separately. Example projections must remain labeled assumptions until measured.

Executive updates fit one page: outcome and current evidence; user/rollout scope; material risk/decision; spend/capacity; next milestone and owner. Engineering appendices hold trace, schema, and benchmark detail. Good communication changes resolution, not truth.

Syllabus checkpoint: the non-negotiable design frame

Before drawing components, write the design contract. Functional requirements describe user-visible behavior and workflows. Non-functional requirements attach measurable constraints to quality, latency, availability, freshness, security, privacy, compliance, recovery, and cost. Make scale estimates for users, tenants, requests, documents/events, vector count, write rate, storage growth, model tokens, and concurrency; show units and peak-to-average assumptions.

Then define API contracts, the data model and ownership of each record, and the end-to-end data flow for ingestion and serving. Walk normal operation plus explicit failure modes: malformed or stale input, duplicate event, dependency timeout, quota exhaustion, partial regional failure, cross-tenant access attempt, model regression, and operator mistake. For each, state detection, containment, recovery, customer behavior, and data reconciliation.

Interview playbook

Use DISCOVER on the whiteboard:

1. D — Desired outcome: users, workflow, source of truth, definition of done, non-goals.
2. I — Inputs and identity: data, ACLs, authority, classification, residency, lifecycle.
3. S — Scale and SLOs: estimates, peaks, quality, latency, freshness, RPO/RTO, budget.
4. C — Contracts and core state: APIs/events, records, state machines, versioning, idempotency.
5. O — Operational architecture: data/control planes, flows, failure isolation, telemetry, capacity.
6. V — Verification and value: evaluations, security tests, reconciliation, outcome baseline.
7. E — Evolution: migration, canary, rollback, cost, rejected alternatives, decision triggers.
8. R — Rollout and responsibility: cohorts, RACI, training, incident/restore, handoff.

Common traps are drawing before asking questions, stating scale without arithmetic, omitting source permissions, treating a vector database as the whole RAG system, trusting the model with tenant identity or authorization, saying “multi-region” without state semantics, ignoring migration/rollback, promising ROI without a baseline, accepting unsafe customer requirements, or presenting only the chosen design without rejected alternatives and reconsideration triggers.

Question bank

These probes test architecture judgment and customer-facing execution together.

Proof artifact: four-design FDE portfolio

Create a reusable portfolio using synthetic scenarios and clearly labeled illustrative estimates. Do not imply customer delivery or personal metrics that are not documented.

1. Choose four timed designs covering different correctness centers: enterprise RAG, multi-tenant assistant/agent platform, document AI or connector, and LLM gateway/evaluation platform.
2. For each, produce a one-page brief: discovery questions, outcomes/non-goals, assumptions with arithmetic, NFRs, APIs/events, core records/state machines, architecture/data flow, trust boundaries, failure table, SLOs, capacity/cost, rollout/rollback, and two rejected alternatives.
3. Build one thin vertical slice for the highest-risk assumption, such as ACL-correct retrieval, resumable approved tool action, generation-based reindex, or policy-consistent model fallback.
4. Create an FDE engagement pack: workshop agenda, assumption/risk register, decision log, pilot definition of done, RACI, migration/cutover checklist, incident/restore drill, executive update, and handoff acceptance.
5. Record a 35-minute whiteboard answer and a two-minute executive version. Review question-first timing, arithmetic, trade-offs, failure recovery, and whether the design maps back to value.

Measure: time to requirements and first architecture, number of explicit assumptions, estimate consistency, critical flows covered, failure/recovery completeness, threat boundaries, rollback units, decision rationale, and communication fit. For the prototype, add task/quality, authorization-negative tests, latency, cost per successful eligible task, and recovery drill time.

Inject failures: tenant mapping error, missed connector event, malformed document, index migration drift, model/provider outage, duplicate tool execution, expired approval, regional loss, and an unsafe stakeholder request. Show designed containment, evidence, decision owner, rollback, and customer communication.

Present: four architecture sheets, live scale calculation, one trace/state-machine demo, option matrix, migration and rollback sequence, risk register before/after the thin slice, pilot scorecard with hypothetical labels, two-minute executive recording, and operator handoff drill.

Chapter review

Architecture depth is the ability to turn uncertain customer needs into testable contracts, quantified trade-offs, controlled authority, recoverable operations, phased change, and durable ownership. FDE depth adds the human system that makes the technical system valuable.

Glossary

Assumption register

A living list of uncertain beliefs, owners, evidence, risk if wrong, and validation status.

Control plane / data plane

The management/provisioning path and the path that processes tenant/user workload data.

Definition of done

Agreed functional, quality/safety, operational, and value/adoption acceptance criteria.

Deployment stamp

A repeatable unit of infrastructure serving one or more tenants to balance isolation and fleet scale.

Forward Deployed Engineering

Customer-facing engineering that discovers, builds, integrates, deploys, and hands off production solutions in context.

Non-goal

An explicit boundary for what a design or release does not attempt to support.

Thin vertical slice

The smallest end-to-end implementation that tests a high-risk assumption across real boundaries.

Rollback unit

The independently reversible version boundary for code, schema, configuration, data, index, or infrastructure.

Mastery checklist

• I begin with outcome, users, authority, data, failure tolerance, and success evidence.
• I calculate illustrative scale transparently and state the benchmark that will replace assumptions.
• I define APIs, events, records, and state machines before naming every component.
• I can compare isolation, index, agent, gateway, connector, pipeline, and evaluation trade-offs.
• I trace security, privacy, observability, cost, migration, failure recovery, and rollback through the design.
• I can state rejected alternatives and objective triggers for reconsideration.
• I can convert an unsafe request into a bounded option and escalate the residual-risk decision.
• I can plan pilot cohorts, value measurement, customer communication, and an operator-tested handoff.

Make staff-level scope visible

A senior answer is not made senior by saying “I led.” It becomes senior when the interviewer can trace how you framed an unclear problem, changed a consequential decision, created leverage beyond your own code, and verified the result.

The six-story portfolio

Prepare one story for each row. A story may cover more than one row, but do not force every question into the same heroic project.

Use STAR-L, but keep the “A” inspectable

Situation gives only the context needed to understand the stakes. Task names your mandate and constraints. Action should consume roughly half the answer: questions asked, analysis performed, options rejected, people aligned, safeguards added, and course corrections. Result separates measured outcomes from impressions. Learning says what you would repeat or change.

Lead decisions, not meetings

Influence without authority comes from improving the decision environment. Make the problem legible, expose trade-offs, invite the right objections, and leave a durable record.

A compact architecture decision record

Title: Choose the retrieval serving path
Status / owner / decision date:
Context: users, scale, sensitivity, current failure
Decision drivers: quality, p95 latency, cost, operations
Options: A / B / C, with evidence and migration effort
Decision: chosen option and why now
Consequences: gains, accepted debt, new risks
Rollout: shadow → limited cohort → broader release
Rollback and revisit triggers:

The record is not a ceremony. It is a compression mechanism: a teammate in another time zone should be able to challenge or execute the decision without reconstructing a meeting.

Resolve disagreement with a ladder

1. Restate the shared objective and the other position until its owner agrees.
2. Classify the disagreement: facts, forecasts, values, constraints, or ownership.
3. Seek disconfirming evidence and define a time-boxed test when the decision is reversible.
4. Ask the accountable owner to decide when evidence cannot remove uncertainty.
5. Disagree, commit, and log a revisit trigger rather than relitigating continuously.

Write so work can continue without you

Current Remote and Sourcegraph role pages explicitly emphasize structured writing, async work, customer communication, autonomy, and substantive review. Treat writing as part of system reliability: missing context creates coordination failures just as missing timeouts create runtime failures. See the live Remote Senior Forward Deployed Engineer role and Sourcegraph Agent Engineer role.

The five-block async update

Outcome: what changed for the user or project
Evidence: test, metric, trace, screenshot, or decision
Risk: what could still invalidate the result
Next: owner and date for the next concrete action
Ask: one explicit decision or help request, if needed

Lead with the outcome, not an activity diary. Replace “worked on evaluation” with “the release gate now catches the three known citation regressions; the multilingual slice is still below its proposed threshold.”

Design memo anatomy

A two-page memo should cover problem and non-goals, users and success measures, constraints and assumptions, proposed flow, alternatives, failure modes, security, operations, rollout, and unresolved questions. Put the recommendation near the top. Attach detailed benchmarks rather than burying the decision beneath them.

Code-review comments that create leverage

Tag the nature of the comment: blocker for correctness or safety, important for maintainability, suggestion for a worthwhile alternative, and nit for optional polish. State the consequence and, where useful, a concrete path forward. Google’s engineering review guide and GitHub’s pull-request review documentation are useful primary references for review practice and review states.

Translate trade-offs for customers

Use consequence language. “A reranker adds another model call” is technical description. “A reranker may improve the difficult queries we sampled, but adds latency and cost to every request; we propose enabling it only for low-confidence queries and measuring both task success and p95 latency” is a decision a customer can evaluate.

Make coding reasoning observable

Senior coding screens still require fundamentals, but the strongest signal is controlled problem solving: establish the contract, choose the simplest correct structure, prove the invariant, and test the boundaries.

The 40-minute loop

1. Clarify: input size, ordering, duplicates, empty values, mutation, return contract, and error behavior.
2. Example: walk one normal case and one adversarial case by hand.
3. Baseline: give a correct simple approach and its time/space cost.
4. Pattern: choose map/set, two pointers, sliding window, stack, heap, binary search, traversal, backtracking, greedy, or dynamic programming because a specific invariant fits.
5. Implement: name state by meaning and narrate only consequential choices.
6. Verify: trace edges, state complexity, and identify what would change at production scale.

Practise the work-shaped screens

The syllabus calls out async Python, SQL, APIs, transformations, tests, unfamiliar code, and reading Go or TypeScript. These exercises reward production instincts more than puzzle tricks.

Async Python: concurrency needs ownership

import asyncio

async def fetch_all(ids, fetch_one):
    async with asyncio.TaskGroup() as group:
        tasks = {item_id: group.create_task(fetch_one(item_id))
                 for item_id in ids}
    return {item_id: task.result() for item_id, task in tasks.items()}

TaskGroup gives the subtasks a lifetime owned by the context. Discuss timeouts, bounded concurrency, partial-result policy, and cancellation cleanup; the Python documentation explains that task-group failure cancels remaining tasks and that cleanup should propagate cancellation after finally work. Read the official coroutines and tasks documentation.

SQL: express the business question first

WITH ranked AS (
  SELECT tenant_id, run_id, cost_usd,
         row_number() OVER (
           PARTITION BY tenant_id ORDER BY cost_usd DESC
         ) AS cost_rank
  FROM agent_runs
  WHERE started_at >= :window_start
)
SELECT tenant_id, run_id, cost_usd
FROM ranked
WHERE cost_rank <= 3;

This asks for the three most expensive runs per tenant, not the three most expensive globally. Explain tie semantics (row_number versus rank), indexes supporting the filter, and why an execution plan matters. Use the official PostgreSQL guides to window functions and EXPLAIN.

API implementation checklist

• Validate request shape and semantic constraints; return stable error contracts.
• Derive identity and tenant scope from authentication, not caller-supplied fields.
• Define idempotency, timeout, retry, concurrency, and transaction boundaries.
• Test happy path, malformed input, duplicate request, dependency timeout, and forbidden access.
• Expose correlation identifiers and useful telemetry without logging secrets.

Debugging unfamiliar code

Restate the symptom, bound the blast radius, reproduce with the smallest input, trace data across boundaries, and form ranked hypotheses. Change one variable at a time. A senior candidate distinguishes mitigation from root-cause correction and adds a regression test plus a detection improvement.

AI-assisted code is still your code

Record the prompt or intent when relevant, inspect every changed line, verify dependencies and licenses, threat-model new data paths, run targeted and broader tests, and be ready to explain the result without the assistant. Current Sourcegraph and Automattic application material asks candidates for concrete opinions about coding agents; speak from a real workflow, while never presenting generated code as independently trustworthy.

Syllabus checkpoint: complete coding-pattern coverage

Keep a compact mental map rather than memorizing isolated solutions. Arrays and strings reward index discipline, maps/sets capture membership and counts, and two pointers or sliding windows exploit ordered or contiguous structure. Stacks model nested or monotonic state; queues model FIFO work; linked lists test pointer ownership and edge cases. Binary search needs a monotone predicate, while interval problems require explicit boundary semantics.

Trees and graphs share traversal machinery but different invariants: tree DFS supports subtree/postorder reasoning, BFS supports levels and minimum unweighted hops, and general graphs require cycle/visited handling. Heaps maintain an extremum or top-k frontier. Backtracking explores a choice tree with pruning; greedy algorithms require an exchange or staying-ahead argument; dynamic programming requires a state that contains all information future choices need. For every pattern, explain correctness, time/space complexity, empty/singleton/duplicate cases, and how you would test it.

Interview playbook

Choose the framework that matches the signal being tested, then keep the answer evidence-led.

Leadership answer: Scope → Judgment → Leverage → Evidence → Learning

1. Scope: users, stakes, constraints, your mandate, and what was ambiguous.
2. Judgment: alternatives and the pivotal decision, including what you rejected.
3. Leverage: how you aligned people or created a pattern others could use.
4. Evidence: measured result, rollout observation, or honest qualitative outcome.
5. Learning: a specific correction you would make next time.

Coding answer: Contract → Baseline → Invariant → Code → Tests → Cost

Clarify the contract, state a correct baseline, name the invariant behind the better approach, implement, trace tests, then give time and space complexity. Mention production concerns only after solving the asked problem.

Common traps

• Using “we” throughout so the interviewer cannot locate personal ownership.
• Claiming consensus when a real decision owner or disagreement existed.
• Giving activity metrics instead of an outcome or risk reduction.
• Narrating every keystroke during coding instead of decisions and invariants.
• Ignoring cancellation, authorization, duplicates, or failure behavior in practical tasks.
• Claiming AI assistance made work faster without showing verification or a quality boundary.

Question bank

Answer aloud. Keep leadership answers near two minutes initially; allow follow-up probes to reveal the deeper evidence.

Proof artifact: the senior-signal packet

Create one reviewable packet that combines leadership, writing, implementation, and explanation. Use a real project only where you can disclose it; otherwise build a clearly labeled sandbox case.

Build it

1. Choose a decision such as adding a reranking stage to a multi-tenant RAG service.
2. Write a two-page design memo and one ADR with criteria, alternatives, security, operations, rollout, rollback, and revisit triggers.
3. Implement a small async API plus a SQL report. Add tests for invalid input, timeout, cancellation, duplicate requests, and cross-tenant access.
4. Request or simulate review; classify comments and record which feedback changed the design.
5. Prepare a two-minute leadership account that states personal ownership without inventing project results.

Measure it

• Memo: decision visible in the first 200 words; every risk has an owner or acceptance statement.
• Code: test pass rate, branch coverage for critical failure paths, static checks, and p50/p95 latency under a declared load.
• Communication: a reviewer can state the decision, main trade-off, and next action after one read.
• Practice: solve and explain the coding task within a fixed time; log clarification, implementation, and verification minutes separately.

If you need numbers before measurement, label them as targets. For example: “hypothetical target: p95 below 800 ms at 20 requests/second,” never as a achieved result.

Inject failure deliberately

Make one downstream call exceed its timeout, send the same idempotency key twice, cancel the client request mid-flight, and attempt a cross-tenant identifier. Verify cleanup, stable error behavior, one durable side effect, and useful trace context. Then seed an AI-generated implementation with a subtle missing tenant predicate and demonstrate that review or tests catch it.

Present it

Bring the memo, ADR, small repository, test report, one trace, review log, and a ten-minute recording. Present the decision in two minutes, code invariant in two, failure evidence in three, and learning in one; reserve two minutes for questions.

Chapter review

Leadership is decision quality plus leverage. Coding is a visible chain from contract to proof. Remote communication is successful when another person can decide or act without recovering missing context.

Glossary

ADR

A short, durable record of an architecture decision, its context, consequences, and revisit conditions.

Invariant

A property that remains true while an algorithm progresses and supports its correctness argument.

Leverage

An intervention that improves the output or judgment of people beyond the author’s own task.

Structured concurrency

A model in which concurrent tasks have explicit ownership and bounded lifetimes.

Revisit trigger

Observable evidence that should cause a prior decision to be re-examined.

Window function

A SQL calculation across related rows while retaining each input row.

Mastery checklist

• I have six distinct, truthful stories with personal actions and defensible evidence.
• I can name the decision owner, rejected option, and revisit trigger in each architecture story.
• I can write an outcome-led update and a two-page memo without a meeting transcript.
• I can distinguish blocker, important, suggestion, and nit review feedback.
• I can solve representative map/window, graph, heap, and basic DP problems while explaining invariants.
• I can reason about async cancellation, SQL ties, API idempotency, authorization, and failure tests.
• I can explain exactly how I verify AI-assisted code.

Compile the role before studying

A job description is neither a complete curriculum nor a keyword list. It is a noisy statement of outcomes, constraints, and organizational anxieties. Your first task is to turn each sentence into an interviewable capability.

Parse five kinds of signal

Make requirements atomic

“Build reliable AI-enabled enterprise integrations” hides at least eight probes: authentication, webhook verification, idempotency, event ordering, tenant isolation, AI evaluation, observability, and rollout. Split it. For each atomic requirement, record one of four evidence states:

• Demonstrated: a real project and artifact can be explained.
• Practised: a sandbox artifact can be shown, clearly labeled as practice.
• Conceptual: trade-offs are understood but implementation evidence is absent.
• Unknown: neither understanding nor evidence is ready.

Live role status snapshot

The supplied syllabus was prepared from selected official pages on 2026-08-03. The following direct checks were repeated on 2026-08-04. “Active” means the official page exposes a named role and application path. “Changed” means the source remains useful but its URL or target role family has moved. “Unavailable” means the supplied role cannot be verified as an open posting; do not prepare or apply as though it were live.

Choose the correct preparation lane

Use the common foundation—production AI, evaluation, backend reliability, security, and communication—but change the center of gravity. The point is not to imitate the job description. It is to surface the most relevant evidence you actually have.

Qdrant FDE: search depth in a customer room

Emphasize: dense/sparse/hybrid retrieval, HNSW, filters and payload indexes, quantization, relevance metrics, capacity, Kubernetes, migration from another search system, and workshop facilitation. Proof: one Qdrant benchmark with a golden set, recall/quality and p95 latency, plus a deployment and rollback diagram. Likely probe: a customer’s filtered search is slow and relevance degraded after migration—how do you isolate data, query, index, resource, and evaluation causes?

Remote FDE: complete enterprise delivery

Emphasize: discovery, APIs, OAuth/service accounts, webhooks, event delivery, idempotency, messy data, RAG/agents, evaluation, multi-tenancy, observability, change management, and reusable patterns. Proof: an enterprise connector and customer reference architecture with definition of done, threat model, golden set, rollout, and reconciliation. Likely probe: turn an imprecise HR or payroll workflow into a secure production plan and show how success will be measured.

Sourcegraph IC4: opinionated agent engineering

Emphasize: multi-step agent reliability, code retrieval and context packing, eval pragmatism, model selection, caching/distillation decisions, cost/latency budgets, technical direction, mentoring, and stack adaptability. The live application asks for hands-on coding-agent experience and where deterministic code or human judgment belongs. Proof: a traceable multi-step code agent, targeted evals, a cost/latency profile, and a two-minute point of view grounded in a real failure.

Canonical alliances architect: breadth with a workshop spine

Emphasize: Linux troubleshooting, DNS/TCP/TLS, Kubernetes, OpenStack, storage, cloud primitives, automation, PostgreSQL/Kafka/NGINX integration, reference architectures, and partner enablement. Proof: a hybrid-cloud reference architecture and a ten-minute workshop segment that explains failure, operations, and cost to mixed audiences. Likely probe: discover a partner environment, make an architecture defensible, and respond thoughtfully when asked outside your deepest specialty.

Supabase AI Platform Engineer: governed internal agents

The current role is a much sharper target than the syllabus’s generic Supabase row. Emphasize: event-triggered execution, durable state, human review, atomic rollback, full run logs, golden suites, CI gates, permission-enforced risk tiers, Python/GCP/infrastructure as code, MCP/API integrations, and value instrumentation. Proof: a registered-agent platform slice in which a forbidden write is impossible at the credential or tool-schema layer, not merely prohibited in a prompt.

Supabase PostgreSQL alternatives: do not confuse adjacency with fit

The active Postgres Engineer role asks for deep internals, extensions in C and Rust, planner/executor/storage mechanics, WAL/MVCC, managed deployment troubleshooting, and large-scale idempotent rollouts. General PostgreSQL, RLS, and pgvector preparation is not equivalent. Mark such requirements honestly as demonstrated, practised, conceptual, or unknown.

Automattic and Deel: preserve signals, discard stale assumptions

The unavailable Automattic posting remains a useful historical prompt for product-first AI, user-facing scale, full-stack breadth, written application quality, and accountable AI-assisted coding. The unavailable Deel posting historically emphasized Node.js, PostgreSQL, AI API integration, ETL, messy data, and document parsing. Do not quote old eligibility, tenure, salary, or location terms as current facts. If a new role appears, recompile it from zero.

Build one evidence pack, then change the lens

A credible evidence pack is a connected body of work, not eight unrelated toy repositories. One enterprise AI system can yield search, evaluation, agent, integration, reliability, security, architecture, and leadership views.

Score gaps by expected interview loss

Use priority = probability of probe × consequence of weakness × improvement per hour. The arithmetic is a forcing function, not scientific precision. A missing must-have with a likely live exercise ranks above an attractive adjacent technology. Eligibility failures rank before preparation: no amount of study fixes an impossible location or schedule constraint.

Numbers without invention

For every real story, prepare the measurement definition, source, time window, baseline, and your contribution. Useful dimensions include corpus or event volume, active users/tenants, quality by slice, p50/p95/p99 latency, cost per successful task, error or duplicate rate, recovery time, delivery lead time, adoption, and customer outcome. If the source metric is inaccessible, use a bounded qualitative statement. Do not convert a hypothetical benchmark into career history.

Run a 48-hour application sprint

Once an active role passes eligibility, stop browsing broadly. Produce a role-specific packet with explicit time boxes.

1. Hour 0–1 — capture: save the official URL, check date, title, location, application fields, hiring stages, and the exact text of consequential requirements.
2. Hour 1–2 — compile: split requirements, label must/preferred/context, assign evidence state, and predict interview formats.
3. Hour 2–4 — select: choose three artifacts and four stories; link every selection to a requirement. Drop weak or duplicative material.
4. Hour 4–6 — tailor: reorder resume bullets and portfolio links without changing facts. Mirror the employer’s problem language only where it accurately describes the work.
5. Hour 6–8 — close one gap: practise the highest-value missing drill: a Qdrant diagnosis, FDE discovery, agent eval, Linux troubleshooting, or governance design.
6. Hour 8–10 — simulate: run a resume deep dive, role-specific technical question, architecture case, and concise written response.
7. Before submission — verify: re-open the page, eligibility, requested format, and AI-use policy. Answer application questions in your own voice and obey any explicit restrictions.

The one-page role brief

Role / official URL / checked date / status:
Eligibility: location, overlap, travel, employment constraints
Top outcomes: 1 / 2 / 3
Must-have capabilities:
Three proof artifacts:
Four stories:
Likely technical and behavioral exercises:
Red gaps and honest framing:
Questions for the interviewer:

Syllabus checkpoint: prepare the numbers behind every role story

For each selected artifact or employment story, prepare a disclosure-safe evidence sheet covering data volume, user and tenant scale, quality change, latency before and after, cost change, reliability or failure-rate change, delivery time, team/stakeholder scope, and customer impact. Record the metric definition, baseline, time window, source, attribution, and confidence. If a dimension was not measured, say so and describe what evidence exists; never fill a blank with a plausible number.

These numbers should remain consistent across resume, application form, recruiter screen, portfolio, and technical loop. Rehearse both an executive statement (“what changed and why it mattered”) and an engineering drill-down (“how measured, what else changed, what failed, and what your contribution was”).

Interview playbook

Answer role-fit questions by joining role evidence to personal evidence without pretending they are identical.

Requirement → Evidence → Judgment → Relevance → Gap

1. Requirement: paraphrase the employer’s real outcome.
2. Evidence: give one truthful project, artifact, or practice example.
3. Judgment: explain the consequential trade-off or failure handled.
4. Relevance: connect it specifically to this environment.
5. Gap: name any meaningful difference and how you would de-risk it.

Role-specific openings

• Qdrant: lead with relevance and performance evidence, then customer delivery.
• Remote: lead with an ambiguous enterprise outcome owned through rollout.
• Sourcegraph: lead with an opinionated agent decision backed by eval, cost, and failure evidence.
• Canonical: lead with breadth, troubleshooting method, and workshop clarity.
• Supabase AI: lead with runtime/evaluation/governance mechanisms, especially structurally enforced permissions.

Common traps

• Keyword recitation with no project decision or artifact.
• Describing a closed role as active because a cached search result still exists.
• Hiding a stack, tenure, location, or schedule gap until late in the process.
• Changing claims between resume, application form, and interview.
• Preparing every target equally and becoming shallow in all of them.

Question bank

These questions test whether targeting is evidence-based rather than cosmetic.

Proof artifact: the live role compiler

Create a versioned dossier for one active role. It should let another reviewer reproduce why you prioritized certain preparation and whether every application claim is supported.

Steps

1. Capture the official posting as a dated PDF or text snapshot for personal analysis, respecting site terms; record the live URL and check time.
2. Extract atomic requirements into a spreadsheet or Markdown file. Tag outcome/domain/context/behavior/eligibility, must/preferred, and evidence state.
3. Link three artifacts and four stories. For each, record exact personal action, source of any metric, disclosure boundary, and one gap.
4. Generate a one-page role brief, six likely technical probes, three questions for the employer, and a 48-hour preparation queue.
5. Have a reviewer compare resume, application answers, dossier, and spoken story for consistency.

Metrics

• 100% of hard eligibility items explicitly resolved as pass, fail, or recruiter question.
• Every must-have mapped to evidence state; no unmarked blanks.
• At least three high-probability requirements supported by inspectable artifacts.
• Every quantitative claim has source, definition, time window, and attribution note.
• A reviewer can explain the top preparation priority and why in under two minutes.

Deliberate failure injection

Replace the saved posting with a closed-role shell, change a location requirement, and insert one unsupported resume claim. Your workflow should flag missing posting metadata, invalidate the prior eligibility decision, and fail the evidence audit. Then simulate a renamed role with the same URL and require a fresh diff rather than silently trusting the old brief.

What to present

Show the dated source register, requirement matrix, evidence links, one-page brief, change diff, and audit result. In an interview, present only your own evidence—not the internal scoring machinery—unless asked how you prepared.

Chapter review

Targeting is disciplined selectivity. Verify the opening, gate eligibility, translate verbs into capabilities, connect truthful evidence, and spend preparation time where it changes likely interview performance.

Glossary

Atomic requirement

One independently assessable capability or constraint extracted from a broader sentence.

Evidence state

Demonstrated, practised, conceptual, or unknown readiness for a requirement.

Eligibility gate

A location, schedule, legal, travel, or hard-experience condition evaluated before deep preparation.

Role compiler

The process that converts a live job description into evidence, gaps, interview hypotheses, and actions.

Historical signal

A useful topic from a closed or changed role that must not be represented as a current requirement.

Evidence provenance

The source, definition, time window, and attribution behind a claim.

Mastery checklist

• I verify title, description, location, and application path—not merely HTTP status.
• I can compile a role into atomic outcomes, depth, context, behavior, and eligibility.
• I know which of the seven syllabus sources are active, changed, or unavailable as of the check date.
• I can name the three strongest artifacts and four stories for my primary active target.
• I distinguish production evidence, sandbox practice, conceptual knowledge, and unknowns.
• I can state stack, tenure, location, travel, and schedule gaps without distortion.
• I re-check the official page and application instructions immediately before submission.

Use an execution operating system

The syllabus proposes two focused hours per day. Protect that constraint: it forces selection, exposes over-engineering, and makes twelve weeks sustainable. A typical five-day week yields ten core hours; keep any sixth session as recovery or mock-interview time rather than silently expanding scope.

The daily 110 + 10 block

The weekly rhythm

• Monday — baseline: define contract, dataset/load, success measure, and simplest working version.
• Tuesday — controlled change: change one consequential variable.
• Wednesday — failure: inject a realistic fault and trace it across the system.
• Thursday — evidence: rerun, compare, analyze slices, and capture limitations.
• Friday — explain: produce a one-page report and a five-to-ten-minute verbal walkthrough.

Before week one, record a baseline mock: one 35-minute architecture, one 30-minute coding problem, one SQL task, and two leadership answers. Score clarification, correctness, depth, failure coverage, and communication from 0–3. This is diagnostic, not a career metric.

Weeks 1–3: retrieval foundations and Qdrant depth

Search comes first because later agent and evaluation work needs a measurable grounding layer. Use one versioned corpus and query set across all three weeks so improvements remain comparable.

Week 1 — establish the retrieval laboratory

Core work: choose a disclosure-safe document corpus; define query intents and judgments; implement lexical, dense, and hybrid retrieval; calculate recall@k, MRR or nDCG as appropriate; capture p50/p95 latency and index size. Include identifier-heavy, semantic, filtered, long-document, and no-answer slices.

Acceptance test: a single command rebuilds the index and produces a report with dataset version, configuration, per-slice quality, latency, and at least five inspected errors. Explain why each metric matches the user task.

Failure: corrupt or omit a subset of relevance judgments. The report must expose dataset coverage rather than quietly comparing incomparable runs.

Week 2 — tune the complete RAG retrieval path

Core work: compare two chunking strategies; query rewrite only where justified; dense/sparse fusion; a reranker; context packing with source boundaries; and incremental indexing. Change one variable per run. Use an error taxonomy such as retrieval miss, wrong rank, filter exclusion, bad chunk boundary, stale data, and correct evidence lost during packing.

Acceptance test: recommend a pipeline for at least two query slices and reject one apparently better aggregate configuration because of latency, cost, or a critical regression. Keep all numbers as measured sandbox results, never employment claims.

Failure: add near-duplicate documents and one stale version. Demonstrate how deduplication, metadata, or recency policy affects citations.

Week 3 — operate Qdrant, not just call it

Core work: model collections, named dense/sparse vectors and payloads; create payload indexes for filters; tune HNSW/search parameters; test quantization; exercise snapshot/restore; and reason about shards, replicas, and multi-tenant layout. Qdrant’s current documentation describes dense+sparse fusion and multi-stage queries in the Hybrid and Multi-Stage Queries guide and memory/performance choices in Quantization.

Acceptance test: publish a before/after matrix for quality, filtered-query p95, throughput, memory or storage, configuration, and workload. Restore from a snapshot into a clean environment and verify document count plus sampled results.

Failure: run a selective filter without the appropriate payload index, then interrupt a migration or restore. Capture symptoms, diagnosis, mitigation, and the safer runbook.

Weeks 4–6: reliable agents, evaluation, and enterprise integration

These weeks join probabilistic behavior to deterministic controls. Use one workflow—for example, an evidence-backed enterprise request triage—so the evaluation and connector are part of the same system.

Week 4 — make the workflow resumable and bounded

Core work: draw explicit states and terminal outcomes; separate deterministic routing, model decisions, tools, and human approval; add schema validation, per-step timeouts, retry budgets, maximum steps, cost budget, checkpointing, idempotent side effects, and trace correlation. LangGraph’s current persistence documentation explains threads, checkpoints, state history, and replay concepts; verify the installed version before coding.

Acceptance test: terminate the process after a checkpoint, restart with the same run identity, and prove that completed side effects are not repeated. Every terminal state must be named: success, rejected, budget exhausted, invalid input, dependency failure, or human cancellation.

Failure: return malformed tool output, make a tool time out, and create a cycle. Show validation, bounded retry, and termination rather than an infinite repair loop.

Week 5 — build an evaluation gate that can say no

Core work: version a golden set; split retrieval, generation, tool-use, task completion, safety, latency, and cost measures; create deterministic assertions; write a human rubric; calibrate any model judge against labeled cases; add slice thresholds and a CI report. Include prompt injection, cross-tenant, no-answer, and malformed-input cases.

Acceptance test: seed a known regression and prove the gate blocks it for the correct reason. Then seed a harmless change and ensure the suite does not fail noisily. Document override authority and the evidence required.

Failure: deliberately bias a judge with order or verbosity and show disagreement against human labels. Tighten rubric or use a deterministic check where possible.

Week 6 — integrate across unreliable boundaries

Core work: design an OAuth/service-account connection, signed webhook ingestion, event queue, idempotent worker, outbox or equivalent handoff, dead-letter path, backfill, replay, and reconciliation. Define API versions, rate limits, tenant scoping, audit events, and long-running job status.

Acceptance test: the same event delivered repeatedly produces one intended side effect; a missed webhook is found by reconciliation; a dead-lettered item can be repaired and replayed with an audit trail.

Failure: crash after the external side effect but before local acknowledgement. Explain why “exactly once” is not a magic transport property and how idempotency plus reconciliation controls the outcome.

Weeks 7–9: data, platform, reliability, and security

Now deploy the same system under realistic operational constraints. The interview goal is to explain what happens after the happy-path demo.

Week 7 — PostgreSQL and replayable ingestion

Core work: normalize the transactional core; use JSONB deliberately; add indexes from query shapes; inspect EXPLAIN (ANALYZE, BUFFERS) safely on test data; exercise transactions, locks, and a deadlock; apply tenant isolation and row-level security; build a checkpointed ingestion path with validation, deduplication, lineage, and quarantine.

Acceptance test: show the slow query, plan, hypothesis, change, new plan, and trade-off. Replay an ingestion partition without duplicating rows. Prove a cross-tenant negative test. PostgreSQL’s official Using EXPLAIN chapter is the primary reference for reading plans.

Failure: feed a malformed PDF-derived record, duplicate an input file, and create lock contention. The pipeline should quarantine or retry without losing lineage.

Week 8 — deploy with explicit lifecycle behavior

Core work: build a small multi-stage image; define configuration and workload identity; deploy the API, worker, and dependencies; set requests/limits; create startup, readiness, and liveness probes with different semantics; add autoscaling assumptions, rolling update, rollback, and a rough cost model. The Kubernetes guide distinguishes how startup, readiness, and liveness probes affect container lifecycle and traffic.

Acceptance test: a slow-starting process is not killed prematurely, an unready instance receives no traffic, a deadlocked process recovers, and a bad release rolls back. State which behavior each probe is designed to observe.

Failure: point readiness at a fragile downstream dependency and observe the amplification. Redesign it to represent whether this instance can serve its contract without creating a fleet-wide outage.

Week 9 — observe, budget, threaten, and recover

Core work: instrument request and agent-step traces, structured logs, metrics, quality samples, token/cost use, and correlation IDs; define user-centered SLIs/SLOs; create actionable alerts; threat-model prompt injection, retrieval poisoning, data leakage, tool abuse, secrets, and cross-tenant access. The OpenTelemetry Python guide provides current instrumentation examples; the OWASP GenAI prompt-injection entry is a useful adversarial checklist.

Acceptance test: use one trace to locate a latency or failure cause, run an incident drill, and produce a short postmortem with detection, mitigation, root cause, correction, and learning. Show preventive and detective controls for a high-risk AI path.

Failure: disable a dependency, exhaust a quota, and place hostile instructions in retrieved content. Verify degradation, bounded retries, human-safe messaging, and alerts tied to action.

Weeks 10–12: convert engineering proof into interview performance

The final phase does not add a new platform. It compresses what you built into timed designs, coding fluency, truthful stories, and role-specific simulations.

Week 10 — four timed system designs

Design an enterprise RAG platform, a governed agent platform, an enterprise connector, and an evaluation platform. Use 45 minutes each: discovery, estimates, APIs/data flow, security/tenancy, failure handling, SLO/observability, cost, rollout/migration/rollback, and rejected alternatives. Record the session and score whether assumptions preceded components.

Acceptance test: each design contains a definition of done, one scale estimate, one critical trust boundary, three failure modes, and a phased rollout. Re-run the weakest design later without reading the first solution.

Failure: have a mock interviewer change a core constraint at minute 20—for example, data cannot leave a private network. Adapt the design without discarding the entire reasoning chain.

Week 11 — coding, SQL, API, and debugging under time

Run mixed sets: maps/windows, stacks/intervals, trees/graphs, heap/greedy, and one basic DP; SQL joins/aggregation/windows/plans; an async API; and an unfamiliar bug. Five high-quality algorithm problems in the week is the syllabus baseline, but quality means a second attempt, edge cases, invariant, and complexity—not merely an accepted submission.

Acceptance test: keep a scorecard for clarification, pattern selection, correctness, tests, complexity, and communication. Re-solve misses from a blank editor after 48 hours. For practical work, include timeout, authorization, idempotency, and failure tests.

Failure: introduce a misleading test, a cancellation bug, and a SQL tie ambiguity. Practise detecting a flawed premise rather than coding around it.

Week 12 — full-loop simulation and application pack

Run at least two role-specific loops: recruiter pitch, resume deep dive, technical discussion, coding/practical task, system design or FDE case, cross-functional scenario, leadership story, and written follow-up. Use the live role compiler from Chapter 10 and re-check availability before each simulation.

Acceptance test: every major topic—search, agents, evaluation, integration, data/platform, reliability, security, customer architecture, leadership—has one credible story or artifact. Gaps are stated, not hidden. The second simulation shows a specific improvement from the first.

Failure: remove a favorite story, challenge a metric’s provenance, and ask for a design in a different domain. The pack should survive without memorized wording or invented precision.

Control the plan instead of obeying it blindly

A sequence is useful only while it targets current constraints. Review the evidence every Friday and change the next week only for a recorded reason.

The weekly scorecard

When time is lost

Do not compress every missed task into the next week. Preserve dependency order and acceptance tests. Drop polish, extra frameworks, frontend work, and duplicate artifacts first. If an interview arrives early, branch into a three-day role sprint: live requirements, top two proof gaps, one mock, then return to the plan.

What not to over-prepare

For these target roles, foundation-model pretraining pipelines, optimizer derivations, distributed training internals, CUDA kernels, state-space-model mathematics, and transformer history have lower expected return unless a live posting explicitly asks. Learn enough to make an informed build/use/fine-tune decision, then return to production retrieval, evaluation, integration, reliability, and architecture.

Interview playbook

Use the twelve-week work as evidence, not as a claim of prior production experience. Label sandbox work explicitly and connect it to real judgment you can defend.

Proof answer: Contract → Baseline → Change → Failure → Evidence → Decision

1. Contract: user task, constraints, and success definition.
2. Baseline: simplest measured system and dataset/load.
3. Change: one variable and the hypothesis behind it.
4. Failure: injected or observed fault and diagnosis.
5. Evidence: quality/latency/cost/operability result with limits.
6. Decision: ship, reject, narrow, or gather more evidence.

Common traps

• Presenting twelve mini-demos with no shared dataset, architecture, or narrative.
• Reporting only aggregate quality and average latency.
• Calling an injected sandbox failure a real production incident.
• Changing multiple variables and attributing the result to one of them.
• Spending the final week polishing slides instead of running mocks.
• Using a library’s feature names as a substitute for understanding lifecycle and failure behavior.

Question bank

Use these as weekly retrospectives and mock-interview prompts.

Proof artifact: the twelve-week evidence repository

Build one repository or portfolio folder that tells a coherent engineering story. Keep sensitive employer material out; use disclosure-safe or synthetic data and label sandbox claims.

Structure and steps

README.md                 # user problem, architecture, how to reproduce
evidence/weekly-log.md    # hypothesis, result, limitation, next decision
retrieval/                # corpus manifest, judgments, benchmark configs
agent/                    # state model, permissions, checkpoints, evals
connector/                # contracts, idempotency, replay, reconciliation
platform/                 # deployment, probes, telemetry, threat model
designs/                  # four timed architecture records
interview/                # story index, scorecards, mock retrospectives

1. Tag a baseline before each controlled change and record dependency versions.
2. Automate one clean setup and one evaluation command; include expected runtime and resource needs.
3. Add a decision ledger that links every claimed improvement to raw evidence and limitations.
4. Record five short walkthroughs: retrieval, agent/eval, integration, operations/security, and architecture.
5. Create a final role index showing which files support which active requirement.

Metrics

• Twelve weekly acceptance tests with pass/fail and evidence link.
• At least one quality, latency, cost/resource, and reliability measure where relevant.
• At least one important slice and one limitation in every benchmark report.
• Time-to-reproduce from a clean environment and time-to-explain in a ten-minute walkthrough.
• Mock score improvement by dimension; do not turn practice scores into employment claims.

Deliberate failure injection

Maintain a failure manifest: missing judgments, stale duplicate documents, slow filtered search, killed agent process, malformed tool output, biased judge, duplicate webhook, malformed ingestion record, lock contention, bad readiness dependency, provider outage, and indirect prompt injection. For each, capture expected behavior, actual signal, containment, repair, regression test, and remaining risk.

What to present

Open with a one-page architecture and evidence map. Demonstrate one clean run and one failure/recovery, then show the decision ledger and a role-specific index. Keep the full repository available for follow-up, but lead with the smallest evidence that answers the interviewer’s question.

Chapter review

The twelve weeks form a dependency chain: measurable retrieval, bounded agents, gates, reliable boundaries, operable deployment, and finally interview compression. The schedule is successful when it produces repeatable evidence and better decisions.

Glossary

Acceptance test

The observable condition that must pass before a week or artifact is considered complete.

Evidence log

A dated record of hypothesis, configuration, result, limitation, decision, and next action.

Controlled change

An experiment that changes one consequential variable while keeping comparison conditions stable.

Failure manifest

A catalog of injected faults, expected behavior, observed evidence, and recovery controls.

Blank-editor reattempt

Solving a missed problem again from scratch after delay to test retained reasoning.

Scope guard

An explicit rule for what is dropped when time or complexity exceeds the plan.

Mastery checklist

• I can state the weekly proof and acceptance test for all twelve weeks.
• My corpus, judgments, configurations, loads, and dependency versions are recorded.
• Every important artifact includes a deliberate failure and recovery evidence.
• I distinguish sandbox results from production experience every time.
• I use role relevance and evidence gaps to adjust the sequence.
• I have four timed designs, a coding/SQL scorecard, six truthful stories, and two full mocks.
• I can reproduce and explain the final repository without hidden manual steps.

Treat job descriptions as volatile production inputs

A role page can change between preparation and submission. Titles move, applicant-tracking systems retain empty shells, location lists narrow, and application questions introduce constraints that were not visible in the main description. The correct response is a source protocol.

Source hierarchy

1. Live official company career page or company-linked ATS: authority for current title, responsibilities, eligibility, and application instructions.
2. Official company handbook and technical documentation: useful for durable work practices and product behavior, but not proof that a requisition is open.
3. Supplied syllabus snapshot: authoritative for this handbook’s intended curriculum, not current availability.
4. Search result, aggregator, repost, or social post: discovery only. Use it to find the official page, never as final authority.

The four-state status model

• Active: named role, substantive description, current location/arrangement, and application path are present.
• Changed: the role or family remains relevant, but title, URL, scope, location, or target opening differs from the syllabus.
• Unavailable: the supplied role redirects to a directory, returns a generic shell, is absent from the current list, or has no usable application path.
• Ambiguous: evidence conflicts; pause role-specific submission and ask the employer or wait for the official system to resolve.

Minimum verification record

company / role / requisition ID:
official URL / final redirected URL:
checked timestamp and timezone:
status and evidence:
locations / overlap / travel / employment type:
must-have outcomes and stack:
application stages and special instructions:
changes from previous capture:
next re-check date / owner:

Official role register: checked 2026-08-04

This register distinguishes live evidence from syllabus history. Re-check it immediately before applying; the date is part of every status statement.

Active roles

Changed target

Supabase careers remains a live official hub, but the syllabus cited a company role family rather than a requisition. On the check date it listed a directly relevant AI Platform Engineer role centered on durable agent execution, human review, rollback, complete logging, evaluation gates, enforced risk tiers, integrations, infrastructure ownership, and value measurement. It also listed a Postgres Engineer role demanding deep database internals and C/Rust extension work. These are different preparation lanes; “Supabase experience” is not a single target.

Unavailable supplied roles

• Automattic — Applied AI Engineer: the supplied role URL redirected to the current jobs directory, and the role was absent from the live embedded listing when checked. Keep product-first AI and async-writing topics as historical syllabus signals, but do not describe the opening as active.
• Deel — Senior Backend Engineer, AI focus: the supplied ATS URL returned a generic jobs shell without named posting metadata. The historic syllabus topics—Node.js, PostgreSQL, AI APIs, ETL, messy data, document parsing—remain useful, but current eligibility and requirements are unverified.

Detect changes that alter preparation

Not every edit matters. Rank changes by whether they affect eligibility, expected interview signal, evidence selection, or the decision to apply.

Diff semantically, not just textually

A reordered paragraph is noise; “remote” changing to a country list is not. Keep a normalized requirement matrix and compare atomic requirements, locations, application questions, and interview stages. Save a dated summary rather than relying on memory. Do not publish copied job-description text; record brief paraphrases and links.

Resolve conflicts conservatively

If the role page says worldwide but the application country selector excludes your location, treat eligibility as ambiguous and ask recruiting before investing heavily. If an old syllabus says eight years and a new requisition says something else, the new official requisition controls. If an aggregator claims active but the official page is a generic shell, mark unavailable.

Audit claims before rehearsing them

The final evidence audit protects both credibility and interview performance. Any claim likely to attract a follow-up must have a source and an explanation of personal scope.

The claim ledger

Use three evidence labels

• Production: work used by real users or operations, described within disclosure limits.
• Practice: a reproducible sandbox or open project built for learning; all numbers are lab measurements.
• Hypothetical: an architecture, target, or worked example not implemented; numbers are assumptions.

Do not let polished presentation blur these categories. A strong practice artifact can demonstrate method, but it does not become production tenure. A hypothetical capacity estimate can demonstrate design reasoning, but it is not an achieved scale.

Red-team the story bank

For each of six leadership stories, ask: Who owned the decision? Who wrote the code? Who measured the result? What did you initially get wrong? What evidence would contradict the story? What can you safely disclose? Remove claims that survive only because no one probes them.

Use role-specific readiness gates

Readiness is not feeling comfortable with every topic. It is the ability to define, design, implement a small version, debug a failure, and explain a production trade-off for the capabilities the active role is likely to test.

The five-evidence ladder

1. Define: explain the concept and why it matters in plain language.
2. Design: place it in a complete system with constraints and alternatives.
3. Build: implement or configure a small reproducible version.
4. Debug: diagnose a realistic failure using evidence.
5. Judge: decide when not to use it and defend the trade-off.

Green, amber, and red decisions

Green: all eligibility gates pass, core must-haves reach at least design/build/debug, and the loop has been simulated. Amber: one material gap can plausibly improve in a short, time-boxed sprint and is framed honestly. Red: eligibility fails, a core must-have is unknown, claims lack provenance, or no complete technical/customer story survives probing. Red means stop or select a different role, not hide the issue.

Full-loop simulation

1. Recruiter pitch and eligibility confirmation.
2. Resume deep dive with metric and ownership probes.
3. Role-depth technical discussion.
4. Coding, SQL, API, or debugging exercise as appropriate.
5. System design or FDE discovery/architecture case.
6. Cross-functional disagreement or customer objection.
7. Leadership and mentoring round.
8. Written follow-up or application response, following the employer’s instructions.

Score each round on clarification, correctness, depth, evidence, trade-offs, and communication. One evaluator should interrupt, change a constraint, and challenge a metric. Improvement between mocks matters more than the first score.

Finish with integrity and a go/no-go decision

Application quality is consistency under scrutiny. Dates, titles, technologies, metrics, ownership, and links must tell the same story in the resume, form, portfolio, and interview.

Submission preflight

• Re-open the official posting and confirm status, requisition, locations, overlap, travel, and deadline.
• Read every application field before drafting; identify own-words, AI-use, confidentiality, and format instructions.
• Check resume links in a logged-out browser and remove private, broken, or misleading artifacts.
• Match every role-specific bullet to the claim ledger and disclosure boundary.
• Use plain, personal language. If assistance is prohibited, do not use generated application content. If assistance is allowed, remain the author and verify every statement.
• Prepare concise questions about outcomes, constraints, team ownership, evaluation, operational responsibility, and interview process.

Decide, do not drift

Interview playbook

Use a final answer structure that makes provenance and relevance easy to inspect.

Claim → Context → Action → Evidence → Limits → Role relevance

1. Claim: make one bounded assertion.
2. Context: users, system, stakes, and your mandate.
3. Action: decisions and work you personally owned.
4. Evidence: result and measurement provenance.
5. Limits: attribution, uncertainty, disclosure, or gap.
6. Relevance: why it transfers to the active role’s outcome.

When the source changed

Say, “I prepared from the current posting checked on [date]. I noticed [specific change], so I adjusted [artifact/story/question].” If the interviewer describes a newer scope, accept the correction, ask clarifying questions, and reason from the new constraints. Do not defend stale notes.

Common traps

• Calling a role active because the URL loads or a search cache has a description.
• Quoting obsolete eligibility or compensation from a closed listing.
• Giving precise impact without metric provenance or personal attribution.
• Presenting a practice lab as a customer deployment or an injected failure as a production incident.
• Using generated application content where the employer requires own words.
• Submitting because preparation time was invested, despite a red eligibility or evidence gate.

Question bank

These drills test source judgment and final readiness. Cite the dated official record when answering status questions.

Proof artifact: the final readiness audit binder

Create a private, versioned binder for one active target. It should prove that the opportunity, claims, evidence, and rehearsal were independently checked. Keep confidential source material out of any public portfolio.

Steps

1. Create the dated source register: official URL, final redirect, named content, application path, constraints, instructions, and a semantic diff from the prior capture.
2. Build the atomic requirement matrix and mark each item demonstrated, practised, conceptual, or unknown.
3. Run the claim ledger across resume, portfolio, four selected stories, and application draft. Remove or qualify unsupported precision.
4. Score the nine readiness domains and run two role-specific full-loop mocks with different interviewers or question sets.
5. Record the go/no-go decision, unresolved recruiter questions, and exact re-check immediately before submission.

Metrics

• 100% of eligibility fields resolved as pass, fail, or explicit question.
• 100% of consequential quantitative claims have provenance and personal-attribution notes.
• Every core must-have has at least one evidence link or a visible gap.
• Every mock round is scored on clarification, correctness, depth, evidence, trade-offs, and communication.
• The second mock corrects at least one named weakness; record the observed change without turning it into a career metric.

Deliberate failure injection

Test the binder by replacing the role page with a generic 200-response shell, changing an eligible location, breaking a portfolio link, removing the source behind one metric, relabeling a sandbox result as production, and adding a prohibited AI-use instruction. The audit should stop the apply-now decision and name every reason.

What to present

The binder itself is mainly private. Present the role-relevant artifact index, clean public proofs, and concise answers with evidence and limits. If asked about preparation, show the readiness matrix and change discipline without exposing confidential records, other applications, or private employer material.

Chapter review

The handbook ends where a real application begins: with a current source, a truthful evidence base, a clear readiness decision, and the ability to adapt when the employer’s needs change.

Glossary

Active

A dated status supported by a named official posting, substantive content, and application path.

Changed

A role or source whose URL, title, scope, location, or target requisition differs materially from the syllabus.

Unavailable

A supplied role that cannot be verified as a current named opening.

Semantic diff

A comparison of meaningful requirements and constraints rather than raw text changes.

Claim ledger

A private register connecting each assertion to context, personal action, evidence, provenance, attribution, and disclosure.

Evidence ladder

Define, design, build, debug, and judge: five progressively stronger readiness signals.

Go/no-go gate

An explicit decision based on availability, eligibility, evidence, integrity, and simulation.

Mastery checklist

• I can justify active, changed, unavailable, or ambiguous status with dated official evidence.
• I do not treat HTTP 200, a cached snippet, or a careers hub as a verified named opening.
• I re-evaluate location, overlap, travel, and application instructions before submission.
• Every resume metric and project claim has provenance, personal scope, and a disclosure decision.
• I label production, practice, and hypothetical evidence without ambiguity.
• I meet the active role’s core evidence gates or have made a clear stop/short-sprint decision.
• I have completed two full-loop mocks and can adapt when a constraint changes.
• I obey role-specific own-words and AI-use instructions.