The AWS GenAI Stack: Bedrock, SageMaker & Serverless AI

1. The AWS GenAI stack as one mental model

Interviewers do not reward service-name recitation; they reward a coherent layering in which every AWS service has a job and an alternative. Hold this model: Amazon Bedrock is the managed model-consumption plane (inference, RAG, agents, safety, evaluation), SageMaker AI is the model-production plane (training, tuning, self-managed serving), and the serverless suite (Lambda, Step Functions, API Gateway, EventBridge, SQS) is the application plane that turns model calls into products. Security, observability, and FinOps primitives cut across all three.

Keep the GCP mirror in your head for portability questions — chapter 09 goes deep on the other side, so here you only need the mapping:

2. Bedrock inference: Converse, throughput modes, and cross-region profiles

Bedrock exposes a per-account, per-region model catalog (Anthropic, Amazon Nova, Meta, Mistral, Cohere, DeepSeek, and others) with access enabled per model. The modern integration surface is the Converse and ConverseStream APIs: one request/response shape across vendors, with system prompts, multimodal content blocks, tool use, and usage metadata. Prefer Converse over the legacy per-model InvokeModel payloads in any design answer — it is what makes model routing and A/B swaps cheap, and it is the seam where an LLM gateway (chapter 07) plugs in.

The decision interviewers actually probe is how you buy tokens. Bedrock has three purchase modes plus a routing layer:

flowchart TD
  A["New Bedrock inference workload"] --> B{"Interactive user traffic?"}
  B -->|"no, latency-tolerant"| C["Batch inference: JSONL in S3, discounted tokens"]
  B -->|"yes"| D{"Steady, predictable token volume?"}
  D -->|"spiky or unknown"| E["On-demand via cross-region inference profile"]
  D -->|"high and steady"| F["Provisioned throughput model units"]
  E --> G["Engineer for throttles: retries with jitter, queue overflow"]
  F --> H["Size units from measured tokens per minute, then commit"]
  C --> J["Outputs feed evals and downstream stores"]
    

Two multipliers change the math on top of any mode. Prompt caching lets you mark cache checkpoints so a stable prefix (system prompt, tool schemas, long documents) is billed at a steeply discounted cache-read rate — on supported models the read discount is on the order of 90% versus fresh input tokens, and time-to-first-token drops because prefill is skipped. Intelligent prompt routing can send easy requests to a cheaper model within a family. Both are useless if your prompt layout churns the prefix on every call — put volatile content (user turn, retrieved chunks that change) after the stable blocks.

3. Knowledge Bases and the vector-store decision

Bedrock Knowledge Bases is managed RAG plumbing: connectors (S3, SharePoint, Confluence, Salesforce, web crawler), parsing (default text extraction or foundation-model parsing for tables and figures), chunking (fixed-size, hierarchical parent-child, semantic, none, or a custom Lambda transform), embedding, and writes into a vector store you choose. At query time you call Retrieve for chunks-plus-scores or RetrieveAndGenerate for a fully managed answer with citations. Chapter 04 covers retrieval science — chunking trade-offs, hybrid search, reranking — so here focus on the AWS-specific decisions: which store, which parsing mode, and when to abandon the managed path.

flowchart LR
  subgraph ING["Ingestion path"]
    S3D["S3 document bucket"] --> KB["Knowledge Base: parse, chunk, embed"]
    KB --> VDB["OpenSearch Serverless vector index"]
  end
  subgraph SRV["Serving path"]
    CL["Client app"] --> GW["API Gateway + Lambda"]
    GW --> ORC["Orchestrator: Retrieve then Converse"]
    ORC --> VDB
    ORC --> FM["Bedrock model (Converse API)"]
    FM --> GRD["Guardrail: grounding check + PII mask"]
    GRD --> GW
  end
  FM --> LOGS["Invocation logging to S3 + CloudWatch"]
    

The senior move is knowing when to leave RetrieveAndGenerate: keep it for internal tools and fast pilots; switch to Retrieve plus your own Converse call the moment you need custom reranking, multi-index federation, query rewriting, or response contracts the managed generator cannot express. That split — managed ingestion, custom generation — is the most common production posture and cites well in interviews (the underlying pattern is the original RAG formulation, Lewis et al. 2020).

4. Bedrock Agents and AgentCore: the agent platform decision

AWS now has two agent stories, and interviewers check whether you know the difference. Bedrock Agents (the original) is a Bedrock-native, prompt-template-driven orchestrator: you define action groups (function or OpenAPI schemas backed by Lambda), attach knowledge bases and guardrails, optionally enable code interpretation and memory, and Bedrock runs the reason-act loop — including return-of-control when you want the client to execute an action. It is fast to stand up and tightly coupled to Bedrock models and its own orchestration style.

Bedrock AgentCore is the 2025-generation answer to a different question: "I already built my agent in LangGraph/Strands/CrewAI with whatever model I want — now give me production infrastructure." It is a set of composable, framework-agnostic services rather than an orchestrator:

• Runtime — serverless execution with per-session isolation and long-running sessions (hours, not API-gateway seconds), so tool-using agents don't inherit request/response time limits.
• Gateway — turns existing APIs and Lambda functions into MCP-compatible tools with auth handled, instead of hand-writing tool adapters per agent.
• Memory — managed short-term session memory and long-term extracted memory shared across sessions.
• Identity — inbound caller auth plus outbound OAuth to third-party services, so agents act with scoped, auditable credentials rather than a god-mode service account.
• Built-in tools — managed Code Interpreter and Browser sandboxes, isolated per session.
• Observability — OpenTelemetry traces of every step into CloudWatch, which is what makes agent debugging tractable.

flowchart TD
  U["Caller: app or workflow"] --> IDN["AgentCore Identity: inbound auth"]
  IDN --> RT["AgentCore Runtime: isolated long-running session"]
  RT --> MEM["AgentCore Memory: session + long-term"]
  RT --> GWY["AgentCore Gateway: APIs and Lambda as MCP tools"]
  GWY --> T1["Lambda tool: order lookup"]
  GWY --> T2["Internal REST API"]
  RT --> CIN["Code Interpreter sandbox"]
  RT --> BRW["Browser tool"]
  RT --> FM["Bedrock models via Converse"]
  RT --> OBS["OTEL traces to CloudWatch"]
    

The third option is no agent service at all: a Step Functions state machine calling Bedrock at each step. That is the right answer more often than vendors admit — when the "agent" is really a deterministic workflow with one or two LLM steps, a state machine gives you replayability, per-step retries, and an audit trail for free, with none of the autonomy risk. Chapter 05 covers agent design patterns themselves; your AWS-specific claim is the placement decision: deterministic flow → Step Functions; dynamic tool choice with production infra needs → AgentCore; Bedrock-native quick build → Bedrock Agents.

5. Guardrails, model customization, and Bedrock evaluations

Bedrock Guardrails is a policy layer evaluated on input and/or output, attachable to direct invocations, agents, and knowledge bases — or callable standalone via the ApplyGuardrail API against any model, including ones outside Bedrock. Know the policy types and, more importantly, what each does and does not catch:

Model customization on Bedrock (docs) spans fine-tuning on labeled pairs, distillation (a larger teacher generates training data to specialize a cheaper student), and custom model import for open weights you tuned elsewhere. The operational fact candidates miss: most customized models must be served on provisioned throughput or dedicated model-copy capacity — so a fine-tune that saves 20% on tokens can lose the business case to an always-on capacity bill. Run the utilization math before recommending tuning; chapters 03 covers when adaptation beats prompting at all.

Close the loop with Bedrock Evaluations: automatic metric jobs, LLM-as-a-judge with your prompt datasets, human-workforce evals, and RAG-specific evaluations that score retrieval and citation quality against a knowledge base. In interviews, position these as the AWS-native execution of the evaluation discipline from chapter 06 — the discipline is portable, the job runner is vendor-specific.

6. SageMaker for GenAI — and the honest self-hosting comparison

SageMaker AI is where you go when Bedrock's catalog or control surface is insufficient. The GenAI-relevant subset: JumpStart for one-click deploy/fine-tune of curated open models; managed training jobs (with spot capacity for interruptible work); real-time endpoints running Large Model Inference (LMI) containers — DJL-Serving images that wrap vLLM/TensorRT-LLM backends with continuous batching, so the chapter-02 serving optimizations arrive pre-packaged; asynchronous inference for large-payload, long-running requests with S3 in/out, an internal queue, and scale-to-zero; and HyperPod for large distributed training — resilient clusters with automated faulty-node replacement and checkpoint-resume, sold on improving goodput (useful training time over wall-clock) for multi-week jobs where a single unhandled hardware fault can cost days.

The interview classic is "Bedrock or self-hosted?" Answer it as a utilization and control question, not a loyalty question:

The senior nuance: this is not a one-time decision. Healthy platforms start on Bedrock for speed, instrument token telemetry from day one, and revisit placement per-workload once volumes stabilize — often landing on a hybrid where one high-volume, narrow task moves to a tuned open model on LMI/EKS while everything else stays managed.

7. The serverless application layer: streaming, orchestration, and queues

Most GenAI system-design failures on AWS happen in the application layer, not the model layer. Three patterns cover nearly every interview scenario.

Streaming chat. API Gateway REST integrations buffer responses and historically capped integrations near 29 seconds (now raisable for regional REST APIs, at a throttling trade-off) — both properties are wrong for token streams. The standard pattern is Lambda response streaming behind a function URL (fronted by CloudFront for auth, WAF, and TLS domain control), forwarding ConverseStream deltas as SSE. WebSockets via API Gateway or AppSync remain the fallback for bidirectional or fan-out cases — chapter 07 covers the protocol-level details.

sequenceDiagram
  participant C as "Client"
  participant F as "CloudFront + function URL"
  participant L as "Lambda with response streaming"
  participant B as "Bedrock ConverseStream"
  C->>F: POST chat turn
  F->>L: forward request
  L->>B: ConverseStream call with tools
  B-->>L: content deltas and tool-use events
  L-->>C: SSE chunks as they arrive
  B-->>L: stop reason plus usage counts
  L-->>C: terminal event with usage
    

Workflow orchestration. Step Functions has optimized Bedrock integrations (synchronous InvokeModel and run-to-completion batch-job steps), per-state retry/backoff/catch, and a distributed map mode that fans out to very high concurrency over S3 objects. Standard workflows give exactly-once-style, auditable state transitions for long processes; Express workflows suit high-rate, short orchestration. This is the backbone for document pipelines and deterministic "agentic" flows.

Event-driven decoupling. EventBridge routes domain events; SQS absorbs bursts in front of workers with per-queue DLQs; every LLM-calling consumer gets bounded concurrency so a traffic spike degrades into queue depth rather than model-quota exhaustion. Chapter 07's idempotency and DLQ discipline applies verbatim — model calls are expensive side effects that must not be replayed blindly.

flowchart LR
  IN["Documents arrive in S3"] --> EVB["EventBridge rule"]
  EVB --> Q["SQS queue with DLQ"]
  Q --> SFN["Step Functions distributed map"]
  SFN --> PRS["Parse: Textract or custom Lambda"]
  PRS --> EXT["Bedrock extraction: batch job or on-demand"]
  EXT --> VAL["Validate: schema + guardrail + confidence"]
  VAL -->|"pass"| OUT["S3 curated zone + DynamoDB metadata"]
  VAL -->|"fail"| REV["Human review queue"]
  OUT --> BI["Athena and QuickSight"]
    

In the batch pipeline, the highest-leverage details are the validation stage (schema-check every model output; route low-confidence extractions to human review rather than averaging them into the lake) and the choice between Bedrock batch inference for large nightly backfills versus on-demand calls inside the map for streaming arrivals. Textract (docs) still beats LLM parsing on cost and determinism for structured forms; use FM parsing where layout understanding genuinely requires it.

8. Security architecture and the Bedrock privacy posture

Regulated-industry interviews are won here. The Bedrock data-privacy posture, per the data protection documentation: prompts and outputs are not used to train base models and are not shared with model providers; inference for a region is processed in that region (or within the profile's geography when you opt into cross-region inference profiles); content is encrypted in transit and at rest. State those four clauses precisely — hand-waving "AWS says it's private" is a junior tell.

flowchart LR
  subgraph VPC1["Customer VPC private subnets"]
    APP["App or Lambda in VPC"]
  end
  APP -->|"PrivateLink, no public internet"| VPE["Interface VPC endpoint bedrock-runtime"]
  VPE --> BRT["Bedrock runtime API"]
  BRT --> GDR["Guardrail policy applied"]
  BRT --> MIL["Invocation logs: KMS-encrypted S3 + CloudWatch"]
  BRT --> TRL["CloudTrail audit trail"]
  APP -.-> ROLE["IAM role: InvokeModel on pinned model ARNs"]
    

• IAM least privilege — scope bedrock:InvokeModel / InvokeModelWithResponseStream to specific model and inference-profile ARNs; separate roles for ingestion, serving, and evaluation; deny wildcard model access in SCPs for regulated accounts.
• Network isolation — interface VPC endpoints for bedrock-runtime and agent runtimes keep invocation traffic off the public internet; endpoint policies restrict which principals and models the endpoint will serve.
• Encryption — customer-managed KMS keys for custom models, knowledge bases, agent sessions, and log destinations; key policy = another audit boundary.
• Audit — CloudTrail for control-plane and API activity; model invocation logging for full request/response bodies to S3/CloudWatch — enabling it is a deliberate compliance decision because prompts often contain the sensitive data itself.
• Guardrails as policy, not the whole defense — prompt-injection resistance also requires tool-permission scoping and retrieval ACLs (chapters 05 and 11).

9. Observability and cost engineering

Bedrock emits CloudWatch metrics per model — invocation counts, latency, input/output token counts, and throttles (monitoring docs). Alarm on throttle rate and p95/p99 InvocationLatency, and trend token counts per feature because tokens are your bill. Invocation logging plus trace IDs from your gateway gives request-level forensics; AgentCore adds step-level OTEL traces for agents. Deeper LLMOps practice — drift, quality regression, incident runbooks — is chapter 11; here, know which AWS surface emits which signal.

Cost engineering is a pricing-dimension inventory plus three levers (Bedrock pricing):

Provisioned-throughput sizing deserves a worked shape (illustrative numbers): if telemetry shows a steady floor of, say, 200k input + 40k output tokens/min during business hours, and one model unit sustains a documented tokens/min ceiling for your model, you buy units to cover the floor and let on-demand (or an inference profile) absorb the spikes above it. Committing to peak instead of floor is the classic overspend; committing before you have per-feature token telemetry is the classic premature optimization. Application inference profiles — invocation profiles you create and tag per workload or tenant — are how spend shows up in Cost Explorer attributable to a team, which chapter 07's per-tenant metering then reconciles.

Fold this into the Well-Architected Generative AI Lens vocabulary when asked "how do you know this is production-ready?": model selection as a reversible decision, safety controls at every trust boundary, evaluation gates before promotion, cost visibility per workload, and operational readiness (quota plans, failover, incident paths). Naming the lens and then demonstrating two of its questions beats reciting all six pillars.

10. Interview scenarios: AWS solution-design drills

These three scenarios cover most senior AWS GenAI loops. Practice narrating each in under four minutes with a drawn diagram.

Scenario 1 — "A bank wants a contact-center assistant grounded in policy documents."

Strong answer skeleton: requirements first (residency, PII exposure, auditability, latency), then Figure 2's shape: S3 + Knowledge Base with hierarchical chunking, OpenSearch Serverless, Retrieve + Converse with a pinned model version, Guardrails with PII masking and contextual grounding, PrivateLink invocation path, invocation logging with CMK, and an eval gate (chapter 06) before any model or prompt change ships. The differentiator is naming what you would refuse: no cross-region inference profile until compliance clears the geography clause; no RetrieveAndGenerate if the bank requires a custom citation contract.

Scenario 2 — "The team has a LangGraph agent prototype; make it production-grade on AWS."

Strong answer skeleton: keep the framework, deploy on AgentCore Runtime for session isolation and long-running executions; move ad-hoc tool code behind Gateway as MCP tools with Identity handling inbound caller auth and outbound OAuth; add Memory instead of a homegrown session store; wire OTEL traces to CloudWatch; put Guardrails on model I/O; add an offline eval harness of recorded trajectories before each release. Flag the alternative honestly: if the graph is actually static, compile it into Step Functions and delete the autonomy.

Scenario 3 — "Process ten million archived contracts and extract obligations monthly."

Strong answer skeleton: Figure 5's pipeline with Bedrock batch inference as the extraction engine (the ~50% discount at this volume is decisive), Step Functions distributed map for orchestration, Textract for the structurally simple pages, JSON-schema validation with confidence thresholds routing to human review, DLQs with replay tooling, and per-run cost reporting via tagged inference profiles. Quantify: estimate tokens/document × corpus size, show the batch-versus-on-demand delta, and state the completion-time trade-off since batch has no latency SLA.

Interview playbook

Answer framework for AWS GenAI questions: (1) restate the workload in capability terms — latency class, volume shape, data sensitivity, autonomy level; (2) place it on the three-plane model (Bedrock consumption / SageMaker production / serverless application); (3) pick services with one named alternative each and a reason; (4) attach the cross-cutting story — IAM, network path, logging, cost attribution; (5) close with the first three production metrics you would watch.

• Senior signals — tokens/min math for throughput decisions; knowing custom models usually need provisioned capacity; the cross-region-profile geography caveat; treating Guardrails as one layer of defense-in-depth; cost per task, not cost per call.
• Common traps — proposing agents for deterministic workflows; defaulting to fine-tuning before RAG and prompting are exhausted; "OpenSearch because it's the default" without the cost floor; API Gateway in front of a token stream; enabling invocation logging without a PII story.
• Stay in your lane — retrieval science lives in chapter 04, agent patterns in 05, evals in 06, streaming protocols in 07, GCP equivalents in 09, LLMOps in 11. Reference them; do not re-derive them mid-answer.

Question bank

Proof artifact: a costed, secured, breakable RAG stack on AWS

Build one small system that produces evidence for five chapter themes at once: inference-mode choice, managed RAG, guardrails, security posture, and cost attribution. Keep the corpus tiny (50–100 public documents) so the whole lab stays in low tens of dollars — and tear down the vector store after, because that is where the idle cost lives.

1. DeployS3 corpus → Knowledge Base → OpenSearch Serverless (or pgvector to compare); serving Lambda using Retrieve + ConverseStream behind a CloudFront-fronted function URL; guardrail with PII masking and contextual grounding attached.
2. HardenPin IAM invoke permissions to exact model ARNs; add a bedrock-runtime VPC endpoint; enable invocation logging to a CMK-encrypted bucket; capture the CloudTrail evidence.
3. MeasureDrive 200 scripted queries; record p50/p95 time-to-first-token, grounding-score distribution, tokens per answer, and cost per answered question from a tagged application inference profile.
4. Break it — throttlingBurst well past your request quota; show the failure mode without retries, then with jittered retries plus an SQS shock absorber; graph both.
5. Break it — groundingInject a contradictory "poison" document into the corpus; show the grounding-check score drop and the blocked/flagged response; discuss what it did not catch.
6. CompareRe-run the query set through batch inference and through a cached-prefix variant; produce a one-page cost table: on-demand vs cached vs batch per 1k answers (label all figures as your measurements, dated).

What to present in an interview: the architecture diagram, the two failure-injection graphs, and the cost table. Three artifacts, each one sentence of setup — "I measured the cache discount myself" is worth more than any certification line on a résumé.

Chapter review

You now hold the AWS GenAI stack as three planes — Bedrock for consumption, SageMaker for production, serverless for application — crossed by security, observability, and cost. The recurring senior pattern: buy tokens deliberately (Figure 1), keep managed ingestion but own generation when contracts demand it (Figure 2), give agents infrastructure only after proving they need agency (Figure 3), stream without buffering hops (Figure 4), industrialize offline work with batch and state machines (Figure 5), and make the private, audited invocation path your default drawing (Figure 6).

Converse API

Bedrock's uniform multi-vendor inference interface with tools, multimodality, and streaming.

Inference profile

Routing identity for invocations; cross-region profiles trade single-region processing for throughput, application profiles carry cost tags.

Provisioned throughput

Dedicated model units with committed tokens/min; the serving floor for most customized models.

Batch inference

Async JSONL-over-S3 jobs at a deep token discount with no completion SLA.

Knowledge Base

Managed RAG ingestion and retrieval: connectors, parsing, chunking, embedding, vector-store writes, Retrieve/RetrieveAndGenerate.

AgentCore

Framework-agnostic agent infrastructure: Runtime, Gateway, Memory, Identity, built-in tools, OTEL observability.

Contextual grounding check

Guardrails policy scoring answer support and relevance against supplied context — a screen, not a proof.

LMI container

SageMaker large-model-inference image wrapping continuous-batching backends like vLLM.

HyperPod

Resilient persistent training clusters engineered for goodput on multi-week distributed jobs.

Model invocation logging

Optional full request/response capture to S3/CloudWatch — an audit asset and a PII liability simultaneously.

• I can choose among on-demand, inference profiles, provisioned throughput, and batch with tokens-per-minute reasoning, and state the cross-region geography caveat.
• I can defend a vector-store choice with a cost floor, an ops-ownership argument, and an ACL story.
• I can place a workload across Step Functions, Bedrock Agents, and AgentCore — and justify refusing agency.
• I can name all Guardrails policy types and two failure modes of contextual grounding checks.
• I can state the Bedrock privacy posture in four precise clauses and design the PrivateLink + KMS + CloudTrail invocation path.
• I can sketch the Bedrock-versus-self-hosted break-even and say what telemetry would trigger a revisit.
• I can cut a Bedrock bill with caching, routing, batch, and right-sized commitments — attribution first, eval gates always.
• I can narrate all three reference architectures from memory in under four minutes each.