CHAPTER 07PRIORITY 1

Reliability, Observability & Security

Define useful telemetry and SLOs, engineer resilience, lead incidents, and defend AI systems against application and enterprise threats.

21 min read Interview drills

Learning objectives

By the end of this chapter, you should be able to:

define user-centered SLIs, SLOs, and an error-budget policy for an AI application;
design logs, metrics, and traces that preserve correlation without leaking sensitive data or exploding cardinality;
combine deadlines, retries, backpressure, bulkheads, load shedding, graceful degradation, and recovery without creating retry storms;
threat-model prompt injection, tool abuse, retrieval poisoning, data exfiltration, and cross-tenant access as system risks;
run a disciplined incident from detection through mitigation, evidence-based root cause, corrective action, and learning; and
produce a failure drill and security artifact that demonstrates production ownership rather than theoretical awareness.

1. Reliability is a user-visible contract

“The pods were up” is not a product outcome. A user needs an authorized, sufficiently correct answer or a truthful, recoverable failure within an acceptable time. Define reliability at that boundary.

SLI, SLO, SLA, and error budget

A service-level indicator (SLI) is a measured ratio or distribution, such as valid successful assistant requests divided by eligible requests.
A service-level objective (SLO) is a target over a window, such as an illustrative 99.5% of eligible requests producing a valid response within a specified latency threshold over 28 days.
A service-level agreement (SLA) is a business/legal commitment and may use different definitions or consequences.
An error budget is the allowed unreliability: for a 99.5% example SLO, 0.5% of eligible events. It becomes useful only when a written policy changes rollout and engineering decisions.

Google’s SRE guidance emphasizes user-centered targets and organizational backing for error-budget consequences: The Art of SLOs. Do not set an SLO from aspiration alone. Examine user tolerance, dependency capability, cost, historical performance, and what action the team will take when the budget burns.

Define eligibility and “good” precisely

User journey	Candidate SLI	Important exclusions or dimensions
Interactive answer	valid, policy-compliant response within latency threshold / eligible requests	Separate user cancellations and invalid auth; slice by tenant tier, region, model route, and request class.
Tool action	confirmed correct terminal outcome / accepted actions	Do not count “model emitted a tool call” as success; distinguish denied, cancelled, compensated, and uncertain.
Document freshness	documents published within freshness target / changed eligible documents	Slice by source, format, tenant, and quarantine reason.
Retrieval quality	evaluated queries meeting grounded-answer threshold / sampled eligible queries	Quality labels arrive slowly; stratify and report uncertainty rather than hiding it in availability.

Use request-based SLIs for interactive traffic and window/backlog-age SLIs for pipelines. For latency, a histogram or event distribution preserves tail behavior; an average can stay healthy while one customer cohort suffers. Define the measurement point and denominator so client disconnects, policy denials, and dependency timeouts cannot be reclassified opportunistically.

Burn rate turns a monthly target into an alert

Burn rate is observed bad-event rate divided by allowed bad-event rate. A service consuming one day’s budget each day burns at 1×. Multi-window alerts combine a short window that detects fast incidents with a longer window that filters transient noise. Page on budget-threatening user impact; ticket on slower trends; dashboard everything else. Exact thresholds depend on the SLO and response model.

2. Observability: connect symptoms to causes

Monitoring asks known questions; observability lets an engineer investigate unanticipated states from system outputs. Instrument a coherent event model, not three disconnected vendors.

Logs, metrics, traces, and context

Metrics aggregate rates, errors, durations, saturation, queue age, token/cost use, and quality samples cheaply enough for dashboards and alerts.
Traces show causality and time across ingress, retrieval, model calls, tools, databases, queues, and policy checks.
Logs explain discrete state transitions and diagnostics with structured fields.
Baggage/context carries selected correlation metadata across process boundaries; it must be size-bounded and must not carry secrets or raw PII.

OpenTelemetry currently defines traces, metrics, logs, and baggage as supported signals: OpenTelemetry signals. Use one resource identity (service.name, version, environment, region), propagate W3C trace context through HTTP and message metadata, and retain application-level IDs such as job or conversation ID in a privacy-safe form.

Trace an AI request without logging the world

request
 ├─ auth + quota
 ├─ retrieval
 │   ├─ embed query
 │   └─ search + ACL filter
 ├─ model generation
 ├─ policy / output validation
 ├─ tool proposal ─ approval ─ tool execution
 └─ response + citations

Record duration, status/error class, model/provider route, token counts, cache outcome, retrieval counts/scores, tool name and outcome, approval decision, validation result, and policy version. Default to hashes, classifications, counts, and references rather than prompts, retrieved text, model output, secrets, or tool arguments. Provide an explicitly authorized, short-retention diagnostic mode for rare cases, with access logs and redaction.

For queued work, the producer span ends before the consumer starts; propagate context in the message and consider a span link when work is batched or fan-outs merge. Trace the retry attempt separately while keeping a shared logical operation ID. Otherwise a three-attempt dependency call looks like one slow span and hides amplification.

Cardinality, sampling, and useful dashboards

Metric label values must remain bounded. model_route or normalized error_class may be useful; user_id, prompt text, document ID, URL, or exception message can create unbounded series and cost. Keep high-cardinality correlation in traces/logs under access control. Use histograms for latency rather than client-side percentile labels; review the official Prometheus histogram guidance for aggregation trade-offs.

Head sampling decides before a trace finishes and is cheap but may miss rare failures. Tail sampling can retain errors, high latency, or important cohorts after observing the trace but requires buffering and collector capacity. OpenTelemetry documents these choices in Sampling. Preserve enough unbiased baseline traffic to estimate rates; an errors-only trace store cannot reveal how unusual an error path is.

A service dashboard should follow user journey → dependencies → resources: SLO and burn, traffic, error classes, latency, quality/freshness, queue age, model/tool outcomes, saturation, and deployment annotations. An alert must say what user promise is threatened, the affected scope, likely first checks, and runbook; if no one should act now, it is not a page.

3. Resilience is a coordinated control loop

Timeouts, retries, breakers, queues, and fallbacks interact. Configure them as a budgeted system; independent defaults often turn one slow provider into a fleet-wide outage.

Deadlines first, then retry

A deadline is the total time the caller is willing to wait. Each downstream timeout must fit inside the remaining deadline and leave time for cleanup or a useful response. Connect, request, stream-idle, and pool-acquisition timeouts protect different waits. Propagate cancellation so abandoned work does not continue spending tokens and database capacity.

Retry only errors likely to improve on another attempt, only when the operation is safe or idempotent, with exponential backoff and full jitter, a capped attempt count, and a shared retry budget. Honor provider rate-limit guidance. If three service layers each retry three times, the lowest dependency may see up to 27 attempts for one user request; choose one owner for retries or coordinate them.

Protection patterns and their limits

Control	Protects against	Failure when misused
Circuit breaker	Repeated calls to a dependency known to be failing	Global breaker hides healthy regions/tenants; probe storms occur in half-open state.
Bulkhead	One dependency or tenant consuming all shared resources	Partitions are too small or unused capacity cannot be borrowed safely.
Backpressure	Producers outpacing consumers	Unbounded queues merely move the outage and increase stale work.
Load shedding	Overload threatening core traffic	Random shedding harms critical traffic; clients retry immediately.
Rate limit/quota	Abuse, runaway cost, and noisy neighbors	A single global quota blocks unrelated tenants; rejected work has no retry guidance.
Fallback	Dependency or model route unavailable	Fallback is untested, lower quality, policy-incompatible, or doubles traffic.

Graceful degradation should preserve truth. Examples: answer from a verified cache with a visible age; switch from an agentic write flow to read-only retrieval; queue a document update and show delayed status; return a cited search result instead of generating; or fail closed for a high-risk tool. A smaller model is not automatically safe: re-run policy and quality gates, disclose capability differences where relevant, and ensure it supports the required region and data terms.

Capacity, disaster recovery, and dependency isolation

Load testing finds throughput and latency under expected mix; stress testing finds the failure boundary; soak testing exposes leaks and slow degradation. Include model latency distributions, streaming connections, long documents, retries, tenant bursts, and cold caches. Capacity plans reserve headroom for failover: if one zone fails, remaining capacity must carry critical load without triggering autoscaling too late.

Set RPO/RTO per state. Conversation records may need point-in-time database recovery; derived vectors may be rebuilt; queued tool actions may require reconciliation before replay; provider credentials may need separate secured recovery. Exercise failover and restore, including the route back to primary. “Multi-region” without conflict, identity, secret, data-residency, and failback design is a diagram, not a recovery plan.

4. AI security: constrain authority, not just text

An LLM processes instructions and untrusted data in the same medium. Prompt rules are useful behavior guidance, but authorization must be enforced by deterministic systems outside the model.

Threat-model the complete flow

List assets (tenant documents, credentials, tool authority, model inputs/outputs, audit evidence), actors (user, tenant admin, insider, compromised document/source, provider, operator), trust boundaries, entry points, and abuse outcomes. Then trace data and authority through retrieval, memory, prompt construction, model, tool broker, downstream API, and logs.

Threat	Preventive controls	Detective/recovery controls
Indirect prompt injection in a retrieved document	Treat retrieved text as data; isolate instructions; least-privilege tools; deterministic authorization; approval for consequential actions	Adversarial evals, tool-policy denials, canary documents, trace/audit review
Excessive agency/tool abuse	Narrow typed tools, user-context credentials, allowlisted parameters, budgets, sandbox, preview/approval	Rate/anomaly alerts, immutable action log, revocation, compensation workflow
Retrieval poisoning	Authenticated ingestion, provenance, version review, publisher trust, ACL at query and fetch	Quality/security scans, lineage lookup, generation rollback and targeted purge
Cross-tenant data exfiltration	Tenant-derived identity, database/index/object isolation, cache-key scoping, output mediation	Cross-tenant tests, access audit, canary tokens, incident deletion workflow
Sensitive output/log leakage	Minimize collection, redact/tokenize, output DLP/policy, retention, provider data controls	Access monitoring, deletion verification, sampled privacy review

The OWASP 2025 LLM guidance identifies prompt injection and excessive agency as distinct but related risks. Its excessive-agency mitigations emphasize minimizing tool functionality, permissions, and autonomy and enforcing downstream authorization: OWASP Excessive Agency. Do not claim input filtering “solves” prompt injection; reduce the impact when the model is influenced.

Tool broker pattern

model proposal → schema validation → policy + user authorization
              → risk/limit check → human approval if needed
              → scoped execution → output validation → audit receipt

The model proposes; the broker decides. Bind each action to authenticated tenant/user, purpose, resource scope, idempotency key, monetary or row limit, and expiry. Resolve opaque resource IDs server-side instead of letting the model supply arbitrary URLs or SQL. High-impact actions receive a human-readable preview based on validated parameters, not free-form model prose. Approval must bind to the exact action digest so parameters cannot change afterward.

Enterprise privacy and evidence

Classify data before choosing model/provider and region. Document data-processing purpose, storage/retention, training/use terms, subprocessors, encryption, residency, deletion, incident handling, and access. Minimize prompts, redact when compatible with the task, and separate customer content from operational telemetry. Rotate secrets and encryption keys through an exercised procedure; never make raw secrets model context.

Audit records should answer who or what principal acted, for which tenant, under which policy/model/tool versions, on which resource, with what approval, outcome, and correlation ID. Avoid storing the sensitive payload when a hash/reference and separately controlled evidence store suffice. Retention and legal requirements vary by customer and jurisdiction; do not turn awareness of GDPR, SOC 2, or ISO 27001 into a claim of compliance. NIST’s Generative AI Profile organizes voluntary risk work across govern, map, measure, and manage: NIST AI 600-1.

5. Worked incident: provider latency becomes a retry storm

This is a hypothetical interview scenario. Example times and measurements illustrate how to tell an incident story; they are not Purnendu’s experience or results.

Detection and mitigation

At 10:02 UTC, fast-burn alerts fire for interactive-answer latency and valid-success SLOs. Queue age and model-call attempts rise, while application CPU remains moderate and provider first-attempt latency rises. A deployment marker shows no internal release. Traces reveal that gateway, orchestration service, and SDK each retry the same timeout, amplifying attempts.

Declare incident, assign incident commander, operations lead, communications lead, and scribe; preserve a shared timeline.
Disable lower-level retries through dynamic configuration, reduce per-request attempt budget, open a route-scoped circuit, and shed noncritical batch traffic.
Route eligible low-risk requests to a previously evaluated fallback; fail closed for unsupported tools and expose a clear retryable status.
Protect recovery by limiting client retry guidance, monitoring fallback capacity/quality, and keeping one controlled probe path to the primary.

By the illustrative 10:18, burn rate falls; by 10:40, the primary is stable but traffic is restored in steps. The team verifies SLO, queue drain, error mix, fallback quality, and cost before resolving. Customer communication states observed impact and current mitigation without speculating about root cause.

Root cause versus trigger

The provider slowdown is the trigger. The internal root cause of severity is uncoordinated retries without a propagated deadline or shared budget, plus a fallback path whose capacity alarm was missing. Contributing conditions include a timeout longer than the upstream request budget, an alert on CPU rather than attempt amplification, and a runbook that did not identify retry owners.

Corrective actions should have owners and verification: one retry layer; deadline propagation test; attempt-count metric; dependency-specific bulkhead; fallback load/quality drill; client retry contract; and a chaos scenario in release qualification. Avoid “be more careful.” A blameless postmortem holds the system and decisions accountable while creating conditions for truthful reporting.

Syllabus checkpoint: operations and enterprise identity

From structured logs to an on-call decision

Structured logs capture discrete, queryable events with stable fields; metrics summarize rates and distributions; traces connect causal work across retrieval, model, tool, queue, and database boundaries. OpenTelemetry provides shared context and export, Prometheus stores/scrapes metrics, and Grafana commonly visualizes and alerts across data sources. Tool choice is secondary to cardinality control, redaction, sampling, retention, and a trace ID that connects the user-visible failure to evidence.

An on-call system needs severity definitions, ownership, escalation, runbooks, safe mitigations, communication cadence, and post-incident follow-through. Alerts should describe a user symptom and an action, not every internal anomaly. Test alert delivery and runbooks during failure drills; an unexercised pager path is not a control.

OAuth/OIDC, IAM/RBAC, and key rotation

OAuth delegates access; OIDC adds an identity layer and ID-token semantics. IAM defines principals and permissions across the platform, while application RBAC maps verified identity to domain roles—often with attribute checks for tenant, resource, or risk. Keep authorization server-side and test denied paths. Key rotation needs overlapping validity, versioned key identifiers, atomic rollout, detection of stale consumers, revocation for compromise, and an audit trail; “replace the secret” is not an operational plan.

Interview playbook

Use the PROMISE framework:

P — Promise: user journey, SLI, SLO, eligibility, window, and policy.
R — Risks: dependency, overload, data, security, privacy, and operator failure modes.
O — Observability: correlation, metrics/logs/traces, sampling, redaction, dashboards, alerts.
M — Mitigation: deadlines, isolation, backpressure, load shedding, safe fallback, and fail-closed cases.
I — Incident: roles, timeline, evidence, communication, and recovery verification.
S — Security: assets, trust boundaries, least authority, deterministic mediation, audit, and retention.
E — Exercise: load/chaos/security tests, restore drills, measured outcomes, and corrective ownership.

Common traps are defining availability at the pod, paging on every error, using raw user IDs as metric labels, storing full prompts by default, stacking retries, treating fallback as merely a cheaper model, claiming prompt injection is prevented by a system prompt, trusting the model to authorize its own tools, listing compliance acronyms as controls, or ending an incident at mitigation without root cause and verification.

Question bank

Practise answers that join reliability, observability, and security rather than treating them as separate teams.

Q1Define an SLO for a retrieval-augmented assistant.

Strong answer outline

Name the journey and eligible events; define good as authorized, valid, sufficiently grounded, and within a latency target.
Separate immediate serving SLI from delayed sampled quality SLI; slice by tenant tier, request class, and route.
Choose target/window from user tolerance and capability, then attach an error-budget policy.

Follow-up probes

Are policy denials errors?
How do you measure answer quality online?

Self-check

Your numerator, denominator, measurement point, slices, and action policy are unambiguous.

Q2Why use multi-window burn-rate alerts instead of a 5% error-rate alarm?

Strong answer outline

Burn normalizes observed bad rate to the SLO’s allowed rate and connects alerts to budget risk.
A short window catches fast incidents; a longer window confirms sustained impact and reduces noise.
Page only actionable threats, with scope/runbook; use slower alerts for gradual consumption.

Follow-up probes

What happens with low traffic?
How do maintenance windows affect eligibility?

Self-check

You tied alerting to user promise and response urgency, not arbitrary percentages.

Q3What telemetry would you capture for one agent tool call?

Strong answer outline

Trace proposal, schema validation, authorization/policy, approval, execution, output validation, and compensation.
Record bounded tool name/version, risk class, outcome/error class, duration, attempt, tenant-safe correlation, and token/cost counts.
Keep arguments/results out by default; use controlled redacted evidence references with retention/access audit.

Follow-up probes

How do retries appear?
What belongs in an audit log versus a trace?

Self-check

You can reconstruct authority and outcome without creating a sensitive shadow dataset.

Q4How do you control metric cardinality in a multi-tenant service?

Strong answer outline

Allow only bounded enumerations such as route and normalized error class; prohibit prompt, document, user, trace, and raw tenant IDs.
Use logs/traces for high-cardinality investigation and aggregate selected tenant cohorts or top-impact reports outside core metrics.
Enforce label allowlists/tests and monitor active series/cost.

Follow-up probes

How do you debug one tenant?
Why are exception messages unsafe labels?

Self-check

You preserve drill-down through correlated signals while bounding the metric dimension space.

Q5Head or tail trace sampling for rare model timeouts?

Strong answer outline

Head sampling is simple and predictable but decides before knowing the outcome.
Tail sampling can retain errors/high latency after completion but needs collector buffering, capacity, and a decision wait.
Use tail rules for rare failures plus an unbiased baseline; monitor dropped telemetry and protect sensitive attributes.

Follow-up probes

How do distributed services make one decision?
Can sampled traces calculate the true error rate?

Self-check

You explain operational cost, statistical bias, and the role of metrics.

Q6How do timeouts and retries become a retry storm?

Strong answer outline

Independent layers exceed the user deadline and multiply attempts while the dependency is already saturated.
Propagate one deadline, assign retry ownership, bound attempts/backoff/jitter, use budgets and idempotency.
Add circuit/bulkhead/admission controls and measure attempts per logical request.

Follow-up probes

What should happen to queued retries after the deadline?
How do clients receive retry guidance?

Self-check

You quantify amplification and coordinate—not merely tune—retry behavior.

Q7Design graceful degradation when the primary model provider fails.

Strong answer outline

Classify requests by capability/risk; use an evaluated compatible fallback only for eligible classes.
Reapply policy, authorization, residency, quality, latency, and cost gates; limit fallback capacity and prevent oscillation.
Fail closed or queue unsupported tool actions, communicate state, and canary restoration.

Follow-up probes

What if fallback output format differs?
How do you prevent doubled spend?

Self-check

Your fallback has a contract, capacity plan, security review, and recovery path.

Q8How would you protect a shared queue from a noisy tenant?

Strong answer outline

Apply authenticated per-tenant admission quota, weighted fair scheduling, concurrency caps, and maximum queue age.
Separate critical interactive and batch pools; bound downstream calls and storage.
Expose tenant lag/rejections with safe retry guidance and tune quotas from contractual capacity.

Follow-up probes

How can unused capacity be borrowed?
What is the failure mode of per-tenant queues?

Self-check

You enforce fairness at admission, scheduling, execution, and dependencies.

Q9Threat-model indirect prompt injection in enterprise RAG.

Strong answer outline

Model a malicious/compromised document crossing ingestion and prompt boundaries to influence tool or output behavior.
Preserve provenance and ACLs; treat retrieved text as untrusted data; mediate typed least-privilege tools with deterministic auth and approvals.
Run adversarial evals, monitor policy/tool denials, audit actions, and support targeted purge/rollback.

Follow-up probes

Why is input sanitization insufficient?
How does retrieval poisoning differ?

Self-check

You reduce blast radius even when model influence succeeds.

Q10How should a human approval gate be secured?

Strong answer outline

Render a deterministic preview from validated typed parameters, actor, target, limits, and expected effect.
Bind approval to an action digest, approver identity, tenant, policy version, expiry, and one-time nonce.
Reauthorize and revalidate at execution; log outcome and support cancellation/compensation.

Follow-up probes

What if the resource changes after approval?
Can the model approve its own request?

Self-check

Your approval cannot be reused or silently altered and does not replace execution-time authorization.

Q11How do you prove a tenant cannot leak through caches and vector search?

Strong answer outline

Derive tenant/security filter from authenticated context and include it in database/index queries and cache namespace.
Fetch-authorize returned source documents, avoid trusting model citations, and isolate administrative paths.
Run property/adversarial tests with identical IDs, crafted filters, stale ACLs, cache collisions, and tenant deletion.

Follow-up probes

Pre-filter or post-filter vector results?
How do ACL changes invalidate caches?

Self-check

You enforce before retrieval, at fetch, and in cache invalidation, with concrete negative tests.

Q12What makes an alert actionable?

Strong answer outline

It maps to threatened user impact/SLO with affected scope and urgency.
It has a responder, likely first checks, relevant dashboards/traces, and a safe runbook action.
It is tested, deduplicated/inhibited appropriately, and reviewed after incidents for precision and recall.

Follow-up probes

Page on CPU at 80%?
How do you detect silent quality regression?

Self-check

You distinguish pages, tickets, and dashboards by required human action.

Q13Tell an incident story when an external provider was the trigger.

Strong answer outline

Quantify user impact and detection; explain roles, timeline, containment, and communications.
Use telemetry to distinguish external trigger from internal severity multipliers such as retries or missing isolation.
Verify recovery and name owned, testable prevention actions and what the team learned.

Follow-up probes

What did you believe initially that was wrong?
How did you avoid unsafe fallback?

Self-check

You demonstrate judgment and system learning without blaming the dependency or inventing metrics.

Q14How do you decide what AI telemetry may be retained?

Strong answer outline

Start from purpose, classification, tenant/customer requirements, provider terms, jurisdiction, and minimum necessary fields.
Prefer derived counts/classes/hashes; segregate sensitive evidence with encryption, access audit, short retention, and deletion propagation.
Document approval, sampling, diagnostic exceptions, and test that logs/traces do not capture secrets or cross tenants.

Follow-up probes

What if debugging needs a prompt?
How do legal holds affect deletion?

Self-check

You balance operational need with explicit governance and do not make unsupported legal claims.

Proof artifact: production ownership drill

Instrument a small RAG-plus-tool service or the chapter 06 platform. Use synthetic data. Any numeric thresholds are example lab objectives.

Define two user journeys with SLI equations, eligibility, example SLOs, error-budget policy, fast/slow burn alerts, and runbooks.
Add OpenTelemetry traces across API, queue, retrieval, model, policy, approval, and tool; export metrics and structured redacted logs. Document attribute allowlists and retention.
Implement deadlines, one retry owner, retry budget, bounded queues, per-tenant bulkhead/quota, circuit breaker, and one evaluated degraded mode.
Create a data-flow threat model and abuse cases for indirect injection, excessive agency, cross-tenant retrieval, secret leakage, and malicious tool output. Add typed tool mediation and an approval digest.
Write an incident template with roles, timeline, impact updates, mitigation decision log, root-cause tree, and corrective-action verification.

Measure: SLI and burn rate, attempts per logical request, p50/p95/p99 latency, queue oldest age, circuit state, shed/degraded requests, fallback quality/cost, trace/log drop rate, active metric series, policy denials, cross-tenant test failures, and recovery time. Keep labels bounded.

Inject failures: disable the model provider; add 2-second latency; return 429s; fill a queue with one tenant; kill the telemetry collector; corrupt a retrieved document with tool instructions; try arbitrary tool parameters; expire approval; leak a fake secret and verify redaction; restore state; and canary recovery. Capture what the user sees and whether the error budget stops release.

Present: SLO sheet, dashboard and alert screenshot, one end-to-end trace, redaction/cardinality tests, threat-model diagram, tool-policy test, incident timeline, root-cause tree, corrective action with owner/test, and a three-minute live dependency-failure drill.

Chapter review

Production ownership connects a measurable promise to telemetry, bounded failure controls, deterministic authority, practiced response, and verified recovery. Reliability and security both reduce uncontrolled blast radius.

Glossary

Burn rate: Observed bad-event rate divided by the rate allowed by an SLO.
Bulkhead: Resource isolation that prevents one workload, dependency, or tenant from exhausting all capacity.
Deadline: The total remaining time within which an operation remains useful to its caller.
Excessive agency: Risk created by giving an AI system more functionality, permission, or autonomy than its task requires.
High cardinality: A label/attribute dimension with many or unbounded distinct values.
Prompt injection: Untrusted content influencing model behavior contrary to the application’s intended instruction hierarchy.
SLI / SLO: A measured service indicator and its target over a defined population and time window.
Tail sampling: Selecting traces after enough of their outcome is known to apply error or latency policies.

Mastery checklist

I can write an SLI equation with an honest denominator and cohort slices.
I can connect error-budget burn to a release and incident policy.
I can trace an AI/tool request without retaining unnecessary sensitive content.
I can bound labels, sampling, retries, queues, concurrency, and fallback capacity.
I can explain which actions degrade, queue, reject, or fail closed.
I can enforce tool authorization outside the model and bind human approval to an exact action.
I can distinguish an incident trigger, root cause, contributors, and verified corrections.
I have exercised restore, dependency failure, cross-tenant access, and prompt-injection scenarios.

Primary sources

Links checked 2026-08-04.