System Design & Forward Deployed Engineering

1. Discovery before diagrams

A Forward Deployed Engineer operates where customer process, messy data, security policy, product capability, and production engineering meet. The first deliverable is a shared problem definition. Drawing components too early hardens guesses into architecture.

Move from request to decision

A customer may ask, “Build an AI assistant for all our policies.” Clarify the job: Who asks which questions? What decision or task follows an answer? Which source is authoritative? Is a citation mandatory? How stale may content be? What must the system refuse? Is the assistant read-only, or may it take actions? What existing workflow, cost, or risk should change?

Write assumptions as tests

Maintain an assumption log with owner, evidence, risk if wrong, and validation date. “Documents contain reliable ACL metadata” becomes: sample 500 representative objects across systems; compare source entitlements with retrieved results; require zero unauthorized returns before pilot. “The model is accurate” becomes an evaluation dataset segmented by task and risk, an acceptance threshold, and human adjudication for disagreement.

Define done at four levels:

1. Functional: supported journeys and explicit non-goals.
2. Quality and safety: groundedness/task success, policy compliance, access control, and human escalation.
3. Operational: latency, availability/freshness, recovery, observability, runbook, support owner.
4. Value/adoption: eligible users, sustained usage, process outcome, and a measurement design that avoids misleading attribution.

2. A repeatable system-design method

Strong system design is structured uncertainty reduction. Use the same sequence in a 45-minute interview and a customer architecture workshop, changing only depth.

Requirements, estimates, contracts, flows

1. Frame: users, core use cases, non-goals, system boundary, source of truth, and success.
2. Quantify: tenants/users, requests per second, concurrency, objects/bytes/vectors, churn, payload size, fan-out, growth, peak factor, latency, SLO, RPO/RTO, and budget range.
3. Define contracts: external APIs/events, job state machines, core records, identity and authorization context, idempotency and versioning.
4. Trace flows: one normal read/write and the highest-risk asynchronous flow, including checkpoints and audit.
5. Choose architecture: components only after their responsibility is clear; identify data/control planes and trust boundaries.
6. Stress: overload, dependency loss, duplicate/out-of-order events, schema/model migration, tenant leak, regional failure, human/operator error.
7. Operate and evolve: SLIs, capacity/cost, deployment, canary, rollback, migration, ownership, and rejected alternatives.

Estimate with ranges, not theatre

Suppose an illustrative design has 50,000 users, 10% active in a peak hour, and six assistant turns per active user: about 30,000 requests/hour or 8.3 requests/second average during that hour. Apply an example 3× burst factor: roughly 25 requests/second. If each request retrieves 20 candidates and sends an average 8,000 total tokens through the model, provider throughput and cost—not API CPU—may dominate. Show the arithmetic, label every assumption, and state which load test or billing sample will replace it.

Capacity is multi-dimensional: requests/second, simultaneous streams, tokens/minute, queue age, database connections, index working set, OCR CPU, provider quotas, and human approval throughput. A system can have idle CPU while blocked on tokens/minute or a saturated connection pool.

Make interfaces carry correctness

POST /v1/assistant/turns
Authorization: Bearer ...
Idempotency-Key: ...
{
  "conversation_id": "...",
  "message": "...",
  "requested_tools": ["policy_lookup"],
  "client_context": {"locale": "en-IN"}
}

The server derives tenant, user, roles, policy and quotas from auth.
The client never supplies a trusted tenant_id or unrestricted tool URL.

Separate synchronous admission from long-running work. Version events and prompts/policies. Give every action and artifact a stable ID, lifecycle state, actor, tenant, provenance, and timestamps. Prefer deterministic downstream authorization over asking the model whether access is allowed.

3. Eight practice architectures and their hard parts

The syllabus’s systems share primitives, but each has a different correctness center. In an interview, spend time where failure is uniquely expensive.

Isolation is a spectrum

A fully shared deployment is cost-efficient and operationally simple but relies heavily on correct logical isolation and fairness. Per-tenant infrastructure improves blast-radius and configuration isolation but costs more and creates fleet-management/version-skew work. Deployment stamps group selected tenants and provide a scalable middle ground. Microsoft’s current multitenancy guidance frames the choice as trade-offs among isolation, cost, scale, performance, complexity, and manageability: architectural approaches for multitenancy.

Decide separately for compute, database, object storage, vector index, queues, encryption keys, network, and model route. A regulated tenant might have a dedicated data plane while sharing a global control plane. The control plane provisions connections, policies, tenants, and deployments; the data plane serves tenant traffic. Compromise or overload of one should not grant authority over the other.

Migrations are systems, too

For a vector-index migration, snapshot a source boundary, bulk backfill into a versioned target, capture concurrent changes via an ordered log/outbox, validate counts and sampled recall/ACL results, shadow or dual-read, canary tenants, then switch a routing pointer. Retain rollback until the old index’s update stream and retention window can safely close. Dual-write alone is not proof: one side can fail silently, so reconciliation is mandatory.

For API, schema, prompt, model, or tool migrations, state compatibility direction and rollback unit. Backward-compatible expand/migrate/contract often beats a flag day. Record the migration version with outputs so evaluation and audit can reproduce behavior.

4. Worked design: a multi-tenant policy assistant with approved actions

This hypothetical system demonstrates a complete interview answer. It is not a claim about Purnendu’s delivered projects or metrics.

Requirements and illustrative scale

Employees ask policy questions with citations and may propose a small set of HR service actions. Answers must honor source permissions and regional retention. High-impact writes require human confirmation. Assume for sizing 200 enterprise tenants, 100,000 total users, 40 peak assistant requests/second, 2 million source documents, a 15-minute freshness target for changed policies, 99.9% illustrative serving availability, and region-specific data planes. Confirm all values in discovery.

Non-goals for the first release: open-ended web browsing, arbitrary SQL or HTTP tools, autonomous approval, payroll decisions, and training on customer content. Success combines evaluated grounded-answer quality by policy domain, zero unauthorized retrieval in adversarial testing, latency/SLO, pilot adoption, and an agreed workflow outcome measured against a baseline.

Architecture

The global control plane maps a tenant to a deployment stamp and manages versioned configuration, but carries no customer prompt content. The regional gateway validates identity, derives tenant/user/roles, applies quotas, and routes only to the mapped data plane. The assistant orchestrator stores a checkpointed turn state with prompt/policy/model/retrieval versions and deadline.

Read and action flows

1. Normalize the user request and run policy/risk classification. Derive authorization filters from identity.
2. Run hybrid lexical/vector retrieval, pre-filter by tenant and ACL where supported, rerank, then reauthorize source fetch. Include provenance and current source version.
3. The LLM gateway selects an allowed model route by capability, region, policy, health, and budget. It enforces deadline/token limits and records usage without raw content by default.
4. Validate the answer structure, citations, policy, and uncertainty. If evidence is insufficient, return a scoped refusal or escalation rather than inventing.
5. For an action, the model emits a typed proposal only. The tool broker reauthorizes the user, validates allowlisted resource IDs and limits, creates an idempotent pending action, and renders a deterministic approval preview.
6. Approval is bound to action digest, approver, tenant, and expiry. Execution uses user-context or narrowly scoped service credentials, then records a receipt. Uncertain remote results enter reconciliation before retry.

Failure, security, and operations

Cost controls include per-tenant/model token budgets, prompt and retrieval limits, cache only where identity/policy/version keys make reuse safe, small-model routing for evaluated classes, incremental ingestion by content hash, and storage lifecycle. Report cost per successful eligible task alongside quality; optimizing cost per raw request rewards cheap failures.

Rejected alternatives

• One model call with all tenant documents: rejected for context limits, cost, stale data, poor provenance, and access-control risk.
• One dedicated stack per tenant from day one: rejected as the universal default because fleet cost and upgrades grow quickly; retained for isolation/residency tiers.
• Let the model call the HRIS directly: rejected because free-form authority, secrets, approval, idempotency, and audit cannot be enforced reliably.
• Active-active global writes immediately: rejected unless recovery requirements justify conflict, replication, residency, and operational complexity.

5. Forward Deployed execution: from pilot to durable ownership

FDE work succeeds when the customer can operate, trust, and extend the system after the initial team leaves. Technical depth and change management are one delivery problem.

Engagement sequence

1. Align: stakeholder map, sponsor, user owner, security/data/platform owners, decision process, RACI, outcomes, constraints, and working cadence.
2. Discover: workflow observation, representative data/ACL sample, architecture/security review, baseline measurement, and assumption/risk register.
3. De-risk: thin vertical slice against the hardest unknown—often access-correct retrieval, malformed documents, tool authorization, or deployment connectivity.
4. Pilot: limited users/data, shadow or read-only mode, explicit acceptance criteria, support channel, daily evidence review, and kill switch.
5. Productionize: SLOs, capacity, threat model, incident/restore drills, runbooks, observability, cost guardrails, and ownership training.
6. Roll out: cohorts with canary metrics, change communication, user education, feedback triage, rollback gates, and decision log.
7. Handoff and expand: architecture record, operations pack, backlog, known limitations, support escalation, outcome review, and next hypothesis.

Migration and workshop artifacts

A useful architecture workshop produces a context/data-flow diagram, requirements and NFR table, identity/ACL map, scale worksheet, risk/assumption register, option matrix, decisions and rejected alternatives, rollout plan, and owners. A migration assessment adds inventory, data quality, dependencies, compatibility, cutover/rollback, parallel-run duration, validation/reconciliation, downtime, training, and decommission criteria.

Do not make the proof of concept a hidden production system. Mark synthetic versus customer data, temporary credentials, retention, unsupported scale, missing controls, and expiry. Promotion requires an explicit review against production criteria rather than enthusiasm.

Handle objections and unsafe requirements

Listen for the underlying need, restate it, and separate non-negotiable safety from negotiable implementation. “No human approval because it slows the workflow” may conceal a latency goal. Offer risk-tiered automation: auto-execute bounded reversible low-risk actions; batch approvals; improve preview UX; keep high-impact irreversible actions gated. Quantify residual risk and seek the authorized risk owner’s decision. Never quietly implement an unsafe exception.

When a request conflicts with product capability or evidence, say: what is known, what is unknown, the failure/blast radius, the recommended safe path, alternatives, and the decision required. Escalation is a delivery skill when it preserves trust and schedule.

Measure value without overselling causality

Choose one primary outcome close to the workflow—such as eligible cases resolved with verified citations or time from accepted request to confirmed completion—and guardrails for quality, safety, cost, and equity across cohorts. Establish baseline and instrument eligibility before rollout. Use phased cohorts or a credible comparison when possible; report adoption and outcome separately. Example projections must remain labeled assumptions until measured.

Executive updates fit one page: outcome and current evidence; user/rollout scope; material risk/decision; spend/capacity; next milestone and owner. Engineering appendices hold trace, schema, and benchmark detail. Good communication changes resolution, not truth.

Syllabus checkpoint: the non-negotiable design frame

Before drawing components, write the design contract. Functional requirements describe user-visible behavior and workflows. Non-functional requirements attach measurable constraints to quality, latency, availability, freshness, security, privacy, compliance, recovery, and cost. Make scale estimates for users, tenants, requests, documents/events, vector count, write rate, storage growth, model tokens, and concurrency; show units and peak-to-average assumptions.

Then define API contracts, the data model and ownership of each record, and the end-to-end data flow for ingestion and serving. Walk normal operation plus explicit failure modes: malformed or stale input, duplicate event, dependency timeout, quota exhaustion, partial regional failure, cross-tenant access attempt, model regression, and operator mistake. For each, state detection, containment, recovery, customer behavior, and data reconciliation.

Interview playbook

Use DISCOVER on the whiteboard:

1. D — Desired outcome: users, workflow, source of truth, definition of done, non-goals.
2. I — Inputs and identity: data, ACLs, authority, classification, residency, lifecycle.
3. S — Scale and SLOs: estimates, peaks, quality, latency, freshness, RPO/RTO, budget.
4. C — Contracts and core state: APIs/events, records, state machines, versioning, idempotency.
5. O — Operational architecture: data/control planes, flows, failure isolation, telemetry, capacity.
6. V — Verification and value: evaluations, security tests, reconciliation, outcome baseline.
7. E — Evolution: migration, canary, rollback, cost, rejected alternatives, decision triggers.
8. R — Rollout and responsibility: cohorts, RACI, training, incident/restore, handoff.

Common traps are drawing before asking questions, stating scale without arithmetic, omitting source permissions, treating a vector database as the whole RAG system, trusting the model with tenant identity or authorization, saying “multi-region” without state semantics, ignoring migration/rollback, promising ROI without a baseline, accepting unsafe customer requirements, or presenting only the chosen design without rejected alternatives and reconsideration triggers.

Question bank

These probes test architecture judgment and customer-facing execution together.

Proof artifact: four-design FDE portfolio

Create a reusable portfolio using synthetic scenarios and clearly labeled illustrative estimates. Do not imply customer delivery or personal metrics that are not documented.

1. Choose four timed designs covering different correctness centers: enterprise RAG, multi-tenant assistant/agent platform, document AI or connector, and LLM gateway/evaluation platform.
2. For each, produce a one-page brief: discovery questions, outcomes/non-goals, assumptions with arithmetic, NFRs, APIs/events, core records/state machines, architecture/data flow, trust boundaries, failure table, SLOs, capacity/cost, rollout/rollback, and two rejected alternatives.
3. Build one thin vertical slice for the highest-risk assumption, such as ACL-correct retrieval, resumable approved tool action, generation-based reindex, or policy-consistent model fallback.
4. Create an FDE engagement pack: workshop agenda, assumption/risk register, decision log, pilot definition of done, RACI, migration/cutover checklist, incident/restore drill, executive update, and handoff acceptance.
5. Record a 35-minute whiteboard answer and a two-minute executive version. Review question-first timing, arithmetic, trade-offs, failure recovery, and whether the design maps back to value.

Measure: time to requirements and first architecture, number of explicit assumptions, estimate consistency, critical flows covered, failure/recovery completeness, threat boundaries, rollback units, decision rationale, and communication fit. For the prototype, add task/quality, authorization-negative tests, latency, cost per successful eligible task, and recovery drill time.

Inject failures: tenant mapping error, missed connector event, malformed document, index migration drift, model/provider outage, duplicate tool execution, expired approval, regional loss, and an unsafe stakeholder request. Show designed containment, evidence, decision owner, rollback, and customer communication.

Present: four architecture sheets, live scale calculation, one trace/state-machine demo, option matrix, migration and rollback sequence, risk register before/after the thin slice, pilot scorecard with hypothetical labels, two-minute executive recording, and operator handoff drill.

Chapter review

Architecture depth is the ability to turn uncertain customer needs into testable contracts, quantified trade-offs, controlled authority, recoverable operations, phased change, and durable ownership. FDE depth adds the human system that makes the technical system valuable.

Glossary

Assumption register

A living list of uncertain beliefs, owners, evidence, risk if wrong, and validation status.

Control plane / data plane

The management/provisioning path and the path that processes tenant/user workload data.

Definition of done

Agreed functional, quality/safety, operational, and value/adoption acceptance criteria.

Deployment stamp

A repeatable unit of infrastructure serving one or more tenants to balance isolation and fleet scale.

Forward Deployed Engineering

Customer-facing engineering that discovers, builds, integrates, deploys, and hands off production solutions in context.

Non-goal

An explicit boundary for what a design or release does not attempt to support.

Thin vertical slice

The smallest end-to-end implementation that tests a high-risk assumption across real boundaries.

Rollback unit

The independently reversible version boundary for code, schema, configuration, data, index, or infrastructure.

Mastery checklist

• I begin with outcome, users, authority, data, failure tolerance, and success evidence.
• I calculate illustrative scale transparently and state the benchmark that will replace assumptions.
• I define APIs, events, records, and state machines before naming every component.
• I can compare isolation, index, agent, gateway, connector, pipeline, and evaluation trade-offs.
• I trace security, privacy, observability, cost, migration, failure recovery, and rollback through the design.
• I can state rejected alternatives and objective triggers for reconsideration.
• I can convert an unsafe request into a bounded option and escalate the residual-risk decision.
• I can plan pilot cohorts, value measurement, customer communication, and an operator-tested handoff.