Enterprise Integrations & Backend Engineering

1. Treat every integration as a changing contract

An enterprise connector is a small distributed system at an organizational boundary: the remote team controls schema, quotas, release cadence, and incident response; your team owns the consequences. Write the contract you need first; isolate the vendor adapter behind it.

REST gives resources, caching, and operational visibility; GraphQL reduces over-fetching but still needs explicit query-cost, pagination, partial-error, and field-authorization handling. Neither removes the need for a canonical internal model: convert remote objects into stable internal types at the edge so provider renames do not spread.

Design the public API around jobs

A sync that may take minutes should not hold an HTTP connection open. Validate authorization and idempotency, persist a job, enqueue it, return 202 Accepted with a status URL exposing queued → running → succeeded | partially_succeeded | failed | cancelled plus counts, timestamps, and a sanitized error summary. Cancellation is observed at safe checkpoints, not instant. Section 7 extends this shape to agent work.

POST /v1/tenants/{tenant_id}/sync-jobs
Idempotency-Key: 7f8f...              # scoped to tenant + operation

202 Accepted
{ "job_id": "job_01...", "status": "queued", "status_url": "/v1/sync-jobs/job_01..." }

Prefer opaque cursors on a deterministic sort key; evolve with contract tests, tolerant readers for additive fields, and telemetry-backed deprecation. “We will version later” is not a strategy.

2. Identity, tenant boundaries, and webhook admission

Authentication proves who is calling; authorization decides what that identity may do; tenant routing decides whose data the action can touch. Keep all three visible.

The IETF baseline — authorization code with PKCE, exact redirect matching, mix-up and CSRF protection, sender-constrained tokens — is RFC 9700, not the original OAuth 2.0 RFC.

Webhook admission pipeline

flowchart LR
  PR["Provider webhook"] --> SZ["Raw-byte size and content-type limit"]
  SZ --> SG["Signature and freshness check"]
  SG --> TN["Tenant lookup by endpoint secret"]
  TN --> IB["Durable inbox insert (unique delivery id)"]
  IB --> AK["Fast 2xx acknowledgement"]
  IB --> WR["Async worker"]
  WR --> DE["Domain event via outbox"]
    

1. Read the exact raw bytes under strict size and type limits; never parse and re-serialize first.
2. Select the secret by authenticated endpoint or connection identifier — never by a tenant ID from the payload.
3. Verify the signature in constant time (GitHub’s guidance is canonical).
4. Enforce a bounded age on signed timestamps and record delivery IDs — signature validity does not prove freshness.
5. Insert metadata and payload hash into an inbox with a unique key, acknowledge fast, process asynchronously.

Accept both secrets during a short audited rotation overlap; record the key version per delivery. Never log tokens, secrets, or unredacted payloads: a correlation ID is useful, a credential is not.

3. Delivery semantics are end-to-end properties

Brokers describe transport behavior; the business outcome also depends on producers, consumers, databases, and external side effects. Say exactly where duplication or loss can occur.

• At-most-once: no retry after uncertainty; work may be lost, duplicates avoided. Only when loss is cheaper than duplication.
• At-least-once: retry until acknowledged; handlers must make duplicates harmless.
• Effectively once: idempotency, uniqueness, and atomic transitions make the result occur once within a defined boundary.

Kafka supports idempotent production and transactions within its own model (delivery-semantics docs), but do not stretch that across an arbitrary email, payroll API, and database. Define the transaction boundary and the compensation path.

Inbox, outbox, and idempotent effects

BEGIN;
INSERT INTO processed_event(tenant_id, event_id)
VALUES (:tenant, :event) ON CONFLICT DO NOTHING;   -- duplicate becomes a no-op
-- Continue only if one row was inserted.
UPSERT employee ...;
INSERT INTO outbox(event_id, aggregate_id, event_type, payload) ...;
COMMIT;

The inbox stops repeated consumption from re-applying a transition. The outbox stores a domain event in the same transaction as the domain change; a relay publishes committed rows. That closes the dual-write gap but not duplicate publication — consumers still deduplicate (AWS transactional outbox guide). For non-idempotent external effects, use a provider idempotency key or a local operation record, and resolve uncertain outcomes by querying the provider before retrying. A client timeout never proves failure.

Ordering, backpressure, and dead letters

Global ordering is expensive and rarely required. Partition by the smallest aggregate needing order — often tenant_id + employee_id — with an aggregate version so consumers reject stale updates. Backpressure is a correctness control: bound worker concurrency, honor rate-limit hints, back off exponentially with full jitter (AWS Builders’ Library), and stop admitting backfills before real-time changes starve. A dead-letter queue is quarantine, not a cemetery: keep error class, attempt history, and a redacted payload reference; replay revalidates authorization and schema, rate-limits release, and audits itself.

4. Streaming LLM responses end-to-end

GenAI adds a transport problem the classic connector never had: a response that streams for tens of seconds, its perceived quality dominated by time-to-first-token (Chapter 2). The job is moving that stream across every hop unbuffered — and defining what the client sees when something dies at token 500.

Choose SSE unless the client must talk during the stream; “stop generation” works as a POST to a cancel endpoint. Disable proxy buffering and compression on the route, flush per event, heartbeat every 10–15 seconds so idle timeouts (60 seconds default on an ALB) do not sever quiet streams; on disconnect, cancel the provider stream — you pay for every generated token, read or not.

sequenceDiagram
  participant C as "Client"
  participant E as "Edge and load balancer"
  participant S as "App service"
  participant P as "Model provider"
  C->>E: POST chat request with stream enabled
  E->>S: forward with deadline
  S->>P: open provider token stream
  P-->>S: token deltas
  S-->>E: SSE events with event ids
  E-->>C: flushed chunks per event
  P-->>S: provider error mid stream
  S-->>C: typed terminal error event then clean close
    

• Mid-stream errors — the 200 left at token one. Emit typed in-band events (delta, error, done) with a mandatory terminal event; an abrupt close without one is an uncertain outcome, not success.
• Backpressure — a slow client cannot pause Bedrock or Vertex. Await socket writes, bound the per-connection buffer, coalesce deltas or cancel at the bound.
• Resumable streams — persist deltas to a durable log keyed by response ID, batched every 50–100 ms; reconnects with Last-Event-ID replay from the offset, then continue live.
• Usage finality — record provider-reported token usage exactly once at finalization, even if the client vanished.

5. The LLM gateway: one front door for every model provider

Once several teams call several models, put a gateway between products and providers — the pattern popularized by LiteLLM-style proxies. Applications speak one internal dialect and reference model aliases; the gateway owns everything a provider swap should not break.

flowchart LR
  APP["Product services"] --> GW["LLM gateway"]
  GW --> ADM["Virtual key auth + budget check"]
  ADM --> CA["Exact and semantic cache"]
  CA -->|"hit"| RES["Serve result"]
  CA -->|"miss"| RTR["Router with model aliases"]
  RTR -->|"primary"| BR["Bedrock"]
  RTR -->|"fallback"| VX["Vertex AI"]
  RTR -->|"fallback"| OA["OpenAI-compatible endpoint"]
  BR --> MET["Token metering + usage events"]
  VX --> MET
  OA --> MET
  MET --> RES
    

Routing is configuration: aliases like fast and reasoning map to provider, model version, and region (data-residency pins live here), with canaries gated by the Chapter 6 evaluation harness. Retries stay in-provider — jittered backoff on 429/5xx inside a latency budget, hedged on time-to-first-token. Fallback crosses providers and is a product decision: tokenizers, context limits, tool schemas, and behavior differ, so only eval-approved pairs enter a chain, responses carry the served model, and a circuit breaker keyed on provider + region + model trips it automatically.

Metering records provider-reported tokens — never your own estimate — with tenant, principal, feature, model, cache flag, latency, and priced cost. Caching starts exact-match: tenant scope, model version, system-prompt hash, normalized parameters, prompt. Semantic caching (embedding similarity over a threshold) multiplies hit rate but risks stale answers and cross-tenant leakage: scope by tenant and ACL hash, exclude personalized or tool-using calls, eval-check served quality. Keep the gateway boring — stateless, horizontally scaled, counters in Redis — it sits on every request path. Bedrock’s cross-region inference and intelligent prompt routing cover slices natively (Bedrock docs); a gateway earns its place once you span providers or need tenant policy.

6. Rate limiting and quotas for token-denominated APIs

Request-per-minute limits fail because one LLM request can cost three orders of magnitude more than another — a 300-token lookup versus a 150k-token document analysis (example figures). Providers meter requests and tokens separately (OpenAI’s rate-limit guide; Bedrock and Vertex quotas are analogous); your platform must do the same per tenant.

The pattern is estimate–debit–reconcile: count prompt tokens exactly, estimate completion from max_tokens or a per-feature historical p95, debit the tenant’s bucket at admission, reconcile against provider-reported usage after. Unbounded max_tokens means unbounded debit — force a cap. Fairness matters more than raw limiting: per-tenant buckets draw weighted shares of provider quota, headroom is reserved for interactive traffic over batch, and a noisy tenant is throttled before the provider throttles everyone.

When rejecting, behave like a good provider: 429 with Retry-After, distinguish exhausted tenant quota from degraded platform capacity, and offer degrade paths — a cheaper alias, truncated context, or conversion to an async job.

7. Async jobs for long-running agent work

Agent runs (Chapter 5 owns the loop) are minutes long and unpredictable — the Section 1 job pattern with more states. Intake validates, persists, enqueues, returns 202; workers lease with a visibility timeout beyond the worst step or heartbeat to extend; each step checkpoints so a crash resumes rather than restarts; cancellation is observed between steps.

stateDiagram-v2
  [*] --> queued
  queued --> running : worker lease
  running --> awaiting_approval : gated tool call
  awaiting_approval --> running : human approves
  awaiting_approval --> cancelled : rejected or expired
  running --> succeeded
  running --> partially_succeeded : some steps failed
  running --> failed : retries exhausted
  queued --> cancelled : cancel requested
  running --> cancelled : observed at checkpoint
  succeeded --> [*]
  partially_succeeded --> [*]
  failed --> [*]
  cancelled --> [*]
    

Deliver status per consumer: polling with backoff and ETags as the baseline; outbound webhooks — you are now the Section 2 provider, owing signed payloads, retries, a DLQ, and redelivery; an SSE status stream for interactive UIs, reusing Section 4’s machinery. For approvals, waits, and compensation, use a durable orchestrator: Step Functions callback task tokens and Workflows callbacks model waiting for a human without a worker burning a lease.

SQS or Pub/Sub for single-step fan-out; Step Functions or Workflows for branches, waits, and compensation; Cloud Tasks for per-queue rate control toward fragile downstreams — Section 3’s admission control, applied outbound.

8. Idempotent AI actions and multi-tenant cost attribution

An agent that sends emails, files tickets, or issues refunds makes Section 3’s idempotency adversarial: the model may phrase the same intent with different text on every retry, so keys must derive from the action, not the words — hash(tenant, job, step, tool, canonical_args).

The effect journal records intent → executing → done-with-result. On any retry — including an LLM retry replaying a step whose tool already ran — the journal answers instead of the tool: the agent sees the recorded result rather than sending a second email. Uncertain outcomes resolve by querying the provider or via provider idempotency keys (Stripe’s design); unrepeatable actions get reserve/confirm phases or a human gate from Figure 4. Classify tools by side-effect risk at registration, not in the prompt.

flowchart LR
  UE["Usage event per call"] --> MT["Metering stream"]
  MT --> EN["Enrich with versioned price sheet"]
  EN --> AG["Hourly rollup per tenant and feature"]
  AG --> BE["Budget engine"]
  BE -->|"soft limit"| DG["Alert and degrade to cheaper alias"]
  BE -->|"hard limit"| HL["Reject new work with 429"]
  AG --> SB["Showback dashboards and invoice export"]
    

Attribution requires every AI-adjacent call to emit a usage event — model calls, embeddings, vector queries, GPU seconds — priced at enrichment from a versioned price sheet so reports survive price changes. Soft budgets alert and degrade the alias; hard budgets reject new work while in-flight streams finish. Reconcile with the invoice monthly: Bedrock application inference profiles and cost-allocation tags on AWS; labels plus the BigQuery billing export on Google Cloud. Chapter 10 covers wider FinOps.

9. Worked system: a resilient HRIS-to-AI knowledge connector

A hypothetical interview-practice system, not a claim about Purnendu’s experience. A customer wants employee directory and policy documents synchronized from an HRIS into an access-controlled AI assistant. Minutes of staleness are acceptable; cross-tenant or cross-group exposure is not.

flowchart TD
  H["HRIS provider"] -->|"OAuth + webhooks"| API["Connector API"]
  API --> IB["Inbox"]
  IB --> Q["Queue"]
  Q --> WK["Bounded workers"]
  WK -->|"paginated reads"| H
  WK --> PG["Canonical PostgreSQL"]
  PG --> OB["Outbox relay"]
  OB --> IX["ACL-aware indexer"]
  SCH["Scheduler"] --> SY["Incremental sync"]
  SY --> WK
  SY --> RC["Reconciliation + audit report"]
    

Key records: connection (tenant, provider, scopes, encrypted credential reference, health), sync_job (cursor, high-water mark, counts), inbox_delivery, source_object (remote ID, version, payload hash, tombstone), employee, outbox_event, sync_anomaly. Tenant-owned keys begin with tenant_id; repositories require tenant context. Sync captures a start high-water mark, pages deterministically, and advances the checkpoint only after durable apply. Webhooks are latency hints; reconciliation compares IDs, counts, and sampled hashes to catch missed webhooks, permission loss, and drift. Deletions become tombstones and removal events.

Python backend judgment

Use async def only for genuinely asynchronous I/O; offload CPU-heavy parsing and blocking SDKs off the event loop (FastAPI’s async guidance). Bound concurrency with a semaphore, pass deadlines through, close clients in lifespan hooks, treat cancellation as control flow.

async with asyncio.timeout(job.remaining_seconds()):
    async with provider_slots:             # protects provider and this process
        page = await client.list_people(cursor=job.cursor)
    await repository.apply_page_atomically(page, job_id=job.id)
    await repository.commit_checkpoint(job.id, page.next_cursor)
# Never catch BaseException and swallow CancelledError.

Validate payloads into strict adapter models (Pydantic), then map to domain types. Unit-test mappings; integration-test transactions and redelivery; contract-test fixtures; end-to-end test a fake provider injecting 429s, timeouts, malformed pages, cursor loops, and duplicate webhooks.

10. Backend breadth checkpoint

Senior interviews probe whether depth in one stack generalizes. The bar: production-strong Python plus the ability to read, implement, and critically review one secondary stack.

• API surface — REST and GraphQL both need authentication, tenant scoping, pagination, stable error contracts, abuse protection; version additively with deprecation telemetry.
• Queues are products — Kafka: partitioned log with replay and offsets. SQS: visibility timeouts, standard/FIFO. Pub/Sub: acknowledged delivery, subscription retention. State ordering, duplication, retention, and DLQ policy per product — never “a message broker.”
• Python service depth — dependency injection for principals, tenant context, sessions; durable work in queues, not fire-and-forget tasks; tests for duplicates, timeouts, cancellation.
• Secondary stack — Node/TypeScript: event loop, promise cancellation, runtime validation. Go: contexts, channels, resource ownership. Same semantics, different spelling.

Interview playbook

Lead with the business invariant, then trace one request and one failure. A compact structure is BOUNDARY:

1. B — Business truth: source of truth, freshness, deletion, conflicts, success metric.
2. O — Ownership and identity: tenant, principal, scopes, data classification, audit actor.
3. U — Uncertainty: timeouts, duplicates, partial failure, ordering, unknown outcomes.
4. N — Normalized contract: canonical model, API/job state, event envelope, versioning.
5. D — Durability: inbox/outbox, checkpoints, idempotency boundary, effect journal, reconciliation.
6. A — Admission control: token-aware limits, budgets, backpressure, deadlines, circuit breakers.
7. R — Recovery and rollout: replay, DLQ, resumable streams, provider fallback, canary tenants.
8. Y — Yardsticks: sync lag, completion rate, duplicates suppressed, cost per tenant, tokens per feature.

Classic traps: “exactly once” without a boundary, email as immutable identity, tenant ID from an unsigned payload, retrying every error, DLQ as recovery. GenAI traps: request-count limits on a token API, SSE through a buffering proxy, no terminal-event protocol, a semantic cache keyed without tenant or model version, silent cross-provider fallback inside an agent loop, tools with no idempotency story. When coding, narrate cancellation, transaction scope, and how a test proves duplicate safety.

Question bank

Practise aloud. Each answer should state assumptions and defend one concrete boundary.

Proof artifact: LLM gateway and resilient connector laboratory

Build a small gateway-plus-connector system against fake providers. All targets are example acceptance thresholds, not claims of prior results.

1. Gateway coreFastAPI proxy over two fake providers with different dialects: virtual keys, model aliases, per-tenant estimate–debit–reconcile buckets, usage events to a metering table.
2. Streaming pathSSE with heartbeats, typed delta/error/done events, Redis delta log with Last-Event-ID resume. Kill a provider mid-stream; demonstrate recovery.
3. Fallback and cacheJittered in-provider retries, an eval-gated cross-provider fallback pair with a circuit breaker, an exact-match cache keyed by tenant + model version + prompt hash.
4. Async agent job202-based job API with the Figure 4 state machine, signed outbound webhooks with retries and DLQ, an idempotent tool-effect journal.
5. Connector spineWebhook inbox, checkpointed sync against a fake HRIS injecting 429s, duplicates, and cursor loops, plus outbox-driven indexing and reconciliation.
6. Cost reportPer-tenant, per-feature showback; a budget breach that degrades the alias, then hard-rejects.

Measure: p50/p95 time-to-first-token through the gateway, stream resume success rate, fallback activations and quality delta, token-estimate error after reconciliation, duplicate effects across 1,000 replayed webhooks and tool retries (lab target: zero), reconciliation drift, cost-report accuracy.

Inject failures: provider dies at token 500; slow client stalls a stream; both providers 429 at once; worker crashes between tool execution and journal write; a webhook replayed 100 times; budget exhausted mid-stream; unknown enum from the HRIS.

Present: a two-minute walkthrough, one client-to-provider trace, the resume demo, before/after reconciliation and cost reports, and a decision record covering SSE versus WebSockets, the fallback gate, and one rejected alternative.

Chapter review

A production integration assumes change, duplication, delay, partial failure, and revoked authority; a GenAI backend adds streams that outlive their status code, costs denominated in tokens, and agents whose retries can repeat side effects. The core is unchanged: explicit contracts, durable state, idempotent effects, bounded and metered work, auditable tenant-aware recovery.

Glossary

Inbox / outbox

Durable tables that deduplicate received messages and atomically stage messages to publish.

High-water mark

Durable boundary showing how far an incremental process has safely advanced.

Reconciliation

Comparing source and destination truth to repair drift event delivery missed.

Last-Event-ID

SSE’s built-in resume cursor: the client replays from the last event it saw.

Terminal event

Mandatory in-band end-of-stream marker; its absence signals an uncertain outcome.

Model alias

Indirection between a capability tier and a concrete provider model version.

Estimate–debit–reconcile

Token-bucket lifecycle: debit an estimate at admission, settle with provider-reported usage.

Effect journal

Durable record of tool-call intent and result that answers retries instead of re-executing.

Showback

Per-tenant, per-feature cost reporting from priced usage events.

Retry budget

Bound on extra attempts so recovery traffic cannot overwhelm a degraded dependency.

Mastery checklist

• I can define an idempotency key’s scope, retention, and concurrency — for API calls and agent tool effects.
• I can trace tenant identity from ingress through queue, database, cache, usage event, and billing tag.
• I can explain why outbox publication and end-to-end exactly-once are different claims.
• I can design an SSE path with heartbeats, in-band terminal events, and Last-Event-ID resume.
• I can defend an eval-gated cross-provider fallback chain and say where it must never trigger.
• I can design token-denominated rate limits with estimation, reconciliation, and tenant fairness.
• I can produce a per-tenant cost report that survives a price change and reconciles against the invoice.