Prompting, Context Engineering & Model Adaptation

1. The adaptation ladder: decide before you tune

The most common senior-level failure here is reaching for fine-tuning to solve a knowledge problem. Fine-tuning reliably changes behavior—format, tone, task framing, tool-selection habits. It is a poor, unauditable way to inject facts, and it cannot keep them fresh: retrieval delivers knowledge with provenance and an update path measured in minutes, while a tuning run's knowledge is frozen and cannot cite a source. Start every adaptation conversation by classifying the gap.

flowchart TD
  A["Capability gap identified"] --> B{"Knowledge gap or behavior gap?"}
  B -->|"knowledge that changes"| C["RAG and retrieval grounding - see Chapter 04"]
  B -->|"behavior, format, tone, task style"| D{"Does prompting plus few-shot pass the eval bar?"}
  D -->|"yes"| E["Ship prompt + context engineering"]
  D -->|"no, and 100s to 1000s of labeled examples exist"| F["PEFT such as LoRA or QLoRA"]
  F --> G{"Quality still short after PEFT sweep?"}
  G -->|"yes, with large data and budget"| H["Full fine-tune of a smaller open model"]
  B -->|"cost or latency gap at acceptable quality"| I["Distill to a smaller student model"]
  H --> I
    

Two criteria interviewers listen for. Data reality: prompting needs zero examples, few-shot 3–10, LoRA is credible from a few hundred vetted pairs, full fine-tuning wants an order of magnitude more. With 40 examples, the answer is prompting plus an eval set, not a training job. Reversal cost: a prompt rollback is a config change; a tuned-model rollback is a deployment with provisioned-capacity implications (Section 8). The ladder is an option-value argument—buy the cheap, reversible option first and let the eval harness (Chapter 06) say when to climb.

2. Prompting as engineering, not incantation

A production system prompt is an API contract: role, capabilities, refusal policy, output format, tool-use policy, and precedence when instructions conflict. Treat it like code—version-controlled, diff-reviewed, regression-tested on every change and model upgrade. The instruction hierarchy matters because user input and retrieved documents are untrusted: system policy outranks user requests, and retrieved content is data, never instructions (injection defense continues in Chapters 05 and 11).

• Structure beats prose — labeled blocks for role, policies, tools, examples, output spec; models follow them more reliably and reviewers can diff them.
• Positive instructions — "respond only with JSON matching the schema" beats piles of "do not" clauses; specify desired behavior plus one fallback (how to abstain).
• Few-shot is bias injection — exemplars anchor format, label distribution, and length; ordering effects are real, so shuffle-test during eval.
• Static vs. dynamic exemplars — per-query nearest-neighbor exemplars lift heterogeneous tasks but destroy cache prefix stability (Section 5); measure whether the lift beats the cache loss.
• Upgrade drift — a prompt is an implicit dependency on a checkpoint; pin versions and gate upgrades on eval regressions, not changelogs.

Chain-of-thought, and when reasoning models retire it

Chain-of-thought prompting (Wei et al., 2022) elicits intermediate reasoning that improves multi-step tasks on models not trained to reason by default. In 2026 the landscape is split: reasoning-first models (OpenAI o-series, Claude extended thinking, Gemini thinking) already run an internal reasoning phase; prepending "think step by step" is at best redundant, at worst interference—and reasoning tokens bill as output even when hidden. The senior position: CoT remains a lever for small or non-reasoning models and for auditable rationales; with reasoning models the lever is the thinking-budget parameter, tuned like any latency/cost knob. Never treat emitted rationales as faithful traces of the computation—they are artifacts, not ground truth.

3. Structured output and tool-schema design

Most enterprise LLM calls are machine-to-machine. Three enforcement tiers, in increasing strength: (1) formatting instructions plus a validator and retry-with-repair; (2) provider "JSON mode," guaranteeing syntactic JSON but not your schema; (3) constrained decoding against a declared schema—OpenAI structured outputs (docs), Gemini responseSchema on Vertex, strict tool schemas on Anthropic and Bedrock—where the sampler masks tokens that would violate the grammar.

Constrained decoding removes parse failures, not semantic failures: the model can emit a schema-perfect object with wrong values, and over-tight schemas can degrade content quality by forcing commitments before evidence. Keep the schema as loose as the consumer allows, make uncertainty representable (nullable fields, an explicit abstain reason), and keep application-level validation as the final authority.

Tool schemas are prompts with types

Function-calling quality is dominated by schema design, because the model chooses tools by reading names and descriptions. Rules that hold across Anthropic tool use (docs), Bedrock Converse, and Vertex function calling:

• Few, sharp tools — selection error grows with catalog size and overlap; merge near-duplicates and say when to use each tool and when not to.
• Enums over free strings — every free-text parameter is a hallucination surface.
• Flat over nested — deep optional nesting multiplies invalid-combination states.
• Declare side effects — read-only vs. mutating drives retries and confirmation gates (Chapter 05).
• Schemas cost tokens every call — a 20-tool catalog can be thousands of prompt tokens; exactly the stable prefix caching amortizes.

4. Context engineering: budgets, rot, compaction, memory

A 200K–2M token window is capacity, not a strategy. Context engineering decides what earns a place in the window, in what order, and what happens when the conversation outlives it. Two empirical failure modes drive the discipline. Lost-in-the-middle: models retrieve best from the beginning and end of long contexts, with a U-shaped accuracy curve over position (Liu et al., 2023). Context rot: performance degrades as the window fills—well below the advertised limit—because distractors dilute attention; needle-in-a-haystack benchmarks are saturated and don't predict this, so test with realistic multi-fact tasks at your operating lengths.

The budget above (illustrative, not a standard) encodes two rules: stable content first, so the prefix is byte-identical across calls and cacheable; volatile content last, for cache mechanics and because the end of context is a high-attention position for the current question. Trigger compaction at an explicit threshold—say 60–70% of the window (a heuristic)—because quality decays before capacity runs out and you must reserve output headroom.

flowchart TD
  H["Conversation history grows"] --> C{"Context usage above budget threshold?"}
  C -->|"no"| ASM["Assemble prompt - stable prefix first, volatile tail last"]
  C -->|"yes"| SUM["Compact older turns into a structured summary block"]
  SUM --> MEM["Persist durable facts and decisions to external memory"]
  MEM --> ASM
  ASM --> CALL["Model call"]
  CALL --> H
    

Compaction is lossy summarization under a contract: preserve decisions, constraints, open questions, and identifiers; drop pleasantries and superseded drafts; log what was dropped—silent compaction is a debugging nightmare. Across sessions, memory splits into a scratchpad (working notes), episodic memory (past sessions, retrieved like RAG), and semantic memory (distilled stable facts, editable and inspectable). Memory retrieval inherits every relevance and access-control problem from Chapter 04—cross-tenant memory leakage is a security incident, and stale memory is a quality bug only eval traces catch.

5. Prompt caching: mechanics and cache-hit economics

Prompt caching stores the computed KV state (Chapter 02) of a prompt prefix so requests sharing that exact prefix skip most prefill compute. It is the highest-leverage cost/latency optimization in prompt-heavy systems—agents with big tool catalogs, long system prompts, fixed-document Q&A. Provider mechanics differ, and the differences are interview material.

sequenceDiagram
  participant C as "Client"
  participant R as "API frontend"
  participant K as "Prefix cache"
  participant G as "GPU prefill"
  C->>R: Request 1 with cache breakpoint after tools
  R->>K: Look up prefix hash
  K-->>R: Miss
  R->>G: Full prefill of entire prompt
  G-->>K: Store KV state for prefix
  G-->>C: Response billed at write rate for prefix
  C->>R: Request 2 with identical prefix plus new question
  R->>K: Look up prefix hash
  K-->>R: Hit within TTL
  R->>G: Prefill only the new suffix tokens
  G-->>C: Faster response with discounted prefix tokens
    

Do the break-even aloud in an interview. With Anthropic-style example rates (1.25× write, 0.1× read), N reuses cost 1.25 + 0.1·(N−1) versus N uncached—caching wins from the second use within the TTL, if the prefix repeats byte-for-byte. That conditional is where systems fail:

• Cache busters — a timestamp, request ID, or user name interpolated early invalidates every prefix; put dynamic content after the last breakpoint.
• Dynamic few-shot — per-query exemplars change the prefix every call; pin per segment or accept the loss knowingly.
• TTL vs. traffic cadence — a 5-minute TTL amortizes in a busy agent session and never hits for hourly visitors; match TTL tier to arrival patterns.
• Hit rate is an SLO — providers return cached-token counts; a deploy that reorders prompt sections can silently zero the hit rate and double the bill. Alert on it.

6. LoRA, QLoRA, and adapter serving

LoRA (Hu et al., 2021) freezes base weights and learns a low-rank update: tuned behavior is W + (α/r)·B·A, with A and B thin matrices of rank r. The bet—empirically sound for behavior shaping—is that task adaptation lives in a low-dimensional subspace. Trainable parameters drop to ~0.1–1%; the artifact is megabytes. Rank r (commonly 8–64) sets capacity; α scales the update, and practitioners tune the α/r ratio rather than either alone. More rank helps until the task's intrinsic dimension is covered, then buys overfitting risk; which modules you adapt (attention projections vs. all linear layers) often moves results more than r.

QLoRA (Dettmers et al., 2023) makes the economics accessible: quantize the frozen base to 4-bit NF4, backpropagate into full-precision adapters, use paged optimizers—the paper fine-tuned a 65B model on a single 48GB GPU. The subtlety to name: you trained against a quantized base, so evaluate in the exact serving configuration you'll deploy; train/serve quantization mismatch is a real regression source (quantized serving is Chapter 02).

flowchart LR
  RQ["Requests tagged with adapter id"] --> SCH["Continuous-batch scheduler"]
  SCH --> BASE["Shared frozen base weights on GPU"]
  subgraph SG1["Adapter pool in GPU and host memory"]
    A1["LoRA - support triage"]
    A2["LoRA - SQL generation"]
    A3["LoRA - claims summaries"]
  end
  SCH -->|"attach per request"| A1
  SCH -->|"attach per request"| A2
  SCH -->|"attach per request"| A3
  BASE --> OUT["Batched decode across tenants"]
  A1 --> OUT
  A2 --> OUT
  A3 --> OUT
    

Multi-LoRA serving usually decides the PEFT-vs-full-FT debate on multi-tenant platforms: systems in the S-LoRA line (Sheng et al., 2023) and engines like vLLM batch requests for different adapters through one shared base, paging adapters between host and GPU memory. Fifty tenant variants become fifty small files on one fleet, not fifty deployments—at the cost of small per-token overhead, cold-swap latency for rare adapters, and a registry mapping tenant → adapter version → base version. That mapping is the sleeper issue: adapters couple to the exact base checkpoint, so a base upgrade is a coordinated retrain-and-reeval event across every adapter.

7. Preference tuning, distillation, and the forgetting problem

SFT teaches what a good answer looks like when a gold answer exists. Preference tuning teaches choosing between plausible answers—helpfulness, tone, safety—where quality is comparative. RLHF as productionized by InstructGPT (Ouyang et al., 2022) trains a reward model on preference pairs, then optimizes the policy with PPO under a KL penalty to the reference. It works and is operationally heavy: a second model to train, RL instability, and reward hacking—the policy exploiting reward-model blind spots such as verbosity and sycophancy.

flowchart LR
  S["SFT checkpoint"] --> G["Sample candidate responses"]
  G --> L["Human or AI preference labels"]
  L --> PP["Preference pairs - chosen vs rejected"]
  PP -->|"RLHF path"| RM["Train reward model"]
  RM --> PO["PPO updates with KL penalty to reference"]
  PP -->|"DPO path"| DL["Direct preference loss - no reward model, no RL loop"]
  PO --> AM["Aligned model"]
  DL --> AM
  AM --> EV["Win-rate evals plus general-capability regression suite"]
    

DPO (Rafailov et al., 2023) collapses the pipeline: a closed-form objective optimizes directly on preference pairs—no reward model, no RL loop, far simpler to reproduce. The trade: no reusable reward model for filtering or online scoring, and quality bounded by the static pair set rather than on-policy sampling. RLAIF—AI feedback replacing human labels, as in Constitutional AI (Bai et al., 2022)—scales label volume while inheriting labeler bias. Interview depth is which knob solves which problem: SFT for task format, preference methods for comparative qualities, DPO as the pragmatic first choice because its failures are visible in data rather than RL dynamics.

Distillation: buying back cost and latency

Distillation transfers a narrow capability from a large teacher to a small student—training on teacher outputs and rationales (Hsieh et al., 2023), or on logits where weights allow. It sits last on the ladder because it presumes you know what good looks like: the teacher sets the ceiling, and the student inherits teacher errors invisibly unless evals cover the tails. Managed offerings (Bedrock Model Distillation; teacher generation plus Vertex tuning) compress the workflow, but the eval obligation stays yours—and distilling proprietary outputs into a competitor model is a licensing question.

Catastrophic forgetting and eval discipline

Every gradient that makes the model better at your task makes it different at everything else. Forgetting shows up as degraded instruction following, lost multilingual ability, or safety drift after a narrow tune—invisible if you only measure the target task. The non-negotiable discipline:

1. Baseline firstFreeze a task eval and a general regression suite (instruction following, safety refusals, benchmark slice); score the base model.
2. Train with mitigationsPrefer PEFT; mix general instruction data into the tuning set; early-stop on validation.
3. Score both suites afterAccept only if task lift clears the bar AND regressions stay within budget; record both in the model card.
4. Canary in servingSmall traffic slice with task metrics and safety monitors (Chapter 11) before ramp.

Synthetic data feeds every rung and needs its own gates: generate with a strong teacher from seeds and real failures; filter with programmatic verifiers then a calibrated LLM judge (Chapter 06); deduplicate; decontaminate against eval sets. A small vetted-human set beats a large unfiltered synthetic dump; the ratio you can defend with an ablation is the senior answer.

8. Cloud mapping: tuning and caching on AWS and GCP

Platform and forward-deployed interviews expect the ladder landed on real services—and the cost-model fine print that changes the recommendation. Full stack tours are Chapters 08 and 09; this is the adaptation-specific mapping.

The portable logic: managed tuning when you want provider models and minimal MLOps; roll-your-own on SageMaker/Vertex custom training for open weights, custom losses (DPO is often DIY territory), multi-LoRA economics, or portability. Either way, eval-before/after is yours—no managed service owns your regression suite.

Interview playbook

For any "should we fine-tune?" question, walk the LADDER:

1. L — Locate the gap: knowledge vs. behavior vs. cost/latency; tuning is for behavior, retrieval for knowledge.
2. A — Attempt the cheapest rung: prompt + context engineering against a frozen eval with a stated pass bar.
3. D — Data audit: vetted example count, labeling ownership, licensing, synthetic-augmentation defensibility.
4. D — Decide with exit criteria: "LoRA if the baseline stalls below X with ≥500 examples; full FT only for deep domain shift on open weights."
5. E — Evaluate before and after: task eval plus general regression to catch forgetting; canary rollout.
6. R — Runtime and cost model: adapter vs. full-copy serving, provisioned floors, cache hit-rate impact, base-upgrade coupling.

Senior signals

• Quantified caching arguments: write premium vs. read discount, TTL vs. cadence, hit rate as a monitored SLO.
• Prompts and schemas as versioned, regression-tested artifacts pinned to model versions.
• Reasoning models move CoT from prompt text to a thinking-budget parameter—with the output-cost implication.
• Naming adapter/base version coupling and provisioned-capacity floors before recommending tuning.

Common traps

• Fine-tuning to "teach the model our docs"—a knowledge/freshness problem retrieval solves with provenance.
• Claiming constrained decoding guarantees correct output—it guarantees shape, not semantics.
• A timestamp at the top of the system prompt, and a mysteriously doubled cache bill.
• Reporting only target-task metrics after tuning, with no regression suite.
• "DPO is better than RLHF" as an absolute rather than a complexity/control trade-off.

Question bank

Questions actually asked for prompting, context engineering, and model adaptation at senior level.

Proof artifact: an adaptation-ladder bake-off

Build a public, reproducible comparison of three rungs on one task—structured extraction from messy support tickets (public or synthetic data). All results are portfolio measurements, not claims about Purnendu's production history.

Steps

1. Freeze the task: extraction JSON schema, 300–500 labeled examples split train/validation/eval, plus a general regression suite (instruction following + safety refusals).
2. Variant A — engineered prompt: versioned system prompt, static few-shot, constrained decoding, validator with one repair retry; iterate to plateau, logging every version's score.
3. Variant B — dynamic few-shot: nearest-neighbor exemplars; measure the accuracy delta AND the cache hit-rate/cost delta vs. A.
4. Variant C — QLoRA tune of an 8B-class open model on the train split via a SageMaker training job or Vertex custom job; serve on a vLLM container with LoRA support.
5. Benchmark identically: per-field accuracy, schema-violation rate, p50/p95 latency, cost per 1,000 requests including cache effects, regression deltas for C.
6. Write a one-page memo recommending a rung at 1K, 50K, and 1M requests/day with break-even arithmetic shown.

Metrics

Per-field precision/recall, exact match, schema-violation and repair-retry rates, cached vs. uncached token counts and TTFT, cost per 1,000 requests per variant, tuning-cost amortization curve, before/after regression scores for the tuned model.

Deliberate failures

• Insert a timestamp at the top of the system prompt; show the cache hit-rate collapse and cost delta.
• Over-tighten the schema and measure content-quality degradation vs. the loose schema.
• Overtrain the QLoRA run (no early stopping) and show the regression suite catching instruction-following decay while task accuracy still looks fine.
• Evaluate the adapter against a differently quantized serving base and document the silent quality drop.

What to present

The ladder diagram with measured numbers per rung, the cost-vs-volume break-even chart, the cache-buster incident graph, and the forgetting demonstration. The arc—"prompting won at low volume, tuning won at 1M/day, here is the crossover"—is exactly the judgment senior interviews probe.

Chapter review

Adaptation is an economics-and-evidence problem. Classify the gap, buy the cheapest reversible option first, and climb only when a frozen eval says the rung below failed. Engineer prompts and schemas as versioned artifacts; order context for stability and cache economics; treat LoRA as the default tuning tool; and never accept a tuned model without before/after scores on both the task and a regression suite.

Glossary

Adaptation ladder

Escalation—prompting, RAG, PEFT, full fine-tune, distillation—governed by gap type, data volume, and reversal cost.

Context rot

Task-quality degradation as the window fills, well below the advertised token limit.

Lost in the middle

U-shaped retrieval accuracy over position in long contexts; beginnings and ends are privileged.

Prompt caching

Reuse of a prefix's computed KV state across requests, discounting cached tokens and cutting TTFT.

LoRA

Frozen base weights plus a trained low-rank update (α/r)·B·A; megabyte-scale artifacts.

QLoRA

LoRA over a 4-bit-quantized frozen base with paged optimizers; single-GPU fine-tuning of mid-size models.

Multi-LoRA serving

Batching requests for many adapters through one shared base model on the same GPUs.

DPO

Direct preference optimization: closed-form loss on preference pairs, no reward model or RL loop.

Catastrophic forgetting

Degradation of general capabilities from narrow fine-tuning; detected only by regression suites.

Constrained decoding

Sampling restricted by a schema/grammar—shape guarantee, not semantic correctness.

Mastery checklist

• I can classify a requirement as knowledge, behavior, or cost/latency and pick the rung with exit criteria.
• I can design system prompts and tool schemas as versioned, regression-tested artifacts.
• I can say when CoT helps, when a thinking budget replaces it, and the billing implication.
• I can order a context window for cache stability and positional attention, with a compaction threshold.
• I can do prompt-cache break-even arithmetic and name the top three cache busters.
• I can explain rank, alpha, and target modules in LoRA and the QLoRA train/serve quantization risk.
• I can design multi-LoRA serving for many tenants and name the base-version coupling.
• I can compare RLHF, DPO, and RLAIF as complexity/control trade-offs.
• I can run eval-before/after and defend a regression budget.
• I can map every rung to Bedrock/SageMaker and Vertex AI, including provisioned-throughput vs. per-token serving.