Enterprise Integrations & Backend Engineering

1. Treat every integration as a changing contract

An enterprise connector is a small distributed system at an organizational boundary. The remote team controls its schema, quotas, release cadence, and incident response; your team owns the consequences. Start by writing the contract you need, then isolate the vendor-specific adapter behind it.

REST, GraphQL, and the canonical model

REST often gives straightforward resource endpoints, caching semantics, and operational visibility. GraphQL can reduce over-fetching and combine related reads, but query complexity, pagination conventions, partial errors, and field-level authorization still need explicit handling. Neither protocol removes the need for a canonical internal model. Convert remote employees, accounts, or opportunities into stable internal types at the edge so provider renames do not spread throughout the product.

Design the public API around jobs

A sync that may take minutes should not hold an HTTP connection open. Accept a request, validate authorization and idempotency, persist a job, enqueue it, and return 202 Accepted with a durable status URL. The status resource should expose a state machine such as queued → running → succeeded | partially_succeeded | failed | cancelled, counts, timestamps, a sanitized error summary, and links to audit details. Cancellation is a request, not an instant fact: workers must observe it at safe checkpoints.

POST /v1/tenants/{tenant_id}/sync-jobs
Idempotency-Key: 7f8f...              # scoped to tenant + operation

202 Accepted
{
  "job_id": "job_01...",
  "status": "queued",
  "status_url": "/v1/sync-jobs/job_01..."
}

For pagination, prefer an opaque cursor tied to a deterministic sort key. For compatibility, use consumer-driven contract tests, tolerant readers for additive fields, and a deprecation process with telemetry showing which clients still use an old version. “We will version later” is not a strategy.

2. Identity, tenant boundaries, and webhook admission

Authentication proves who is calling; authorization decides what that identity may do; tenant routing decides whose data the action can touch. Keep all three visible in the design.

Choose the principal deliberately

The current IETF OAuth security best practice recommends authorization code flows with PKCE, exact redirect-URI matching, protection from mix-up and CSRF attacks, and sender-constrained tokens where applicable; it also deprecates insecure legacy patterns. Use this as the baseline rather than treating the original OAuth 2.0 RFC as the last word: RFC 9700.

Webhook admission pipeline

1. Read the exact raw bytes with a strict size and content-type limit. Do not parse and re-serialize before verification.
2. Select the secret by authenticated endpoint or connection identifier, not by a tenant ID trusted from the payload.
3. Verify the provider’s HMAC or asymmetric signature using a constant-time comparison. GitHub’s current guidance, for example, signs the payload and recommends secure comparison: validating webhook deliveries.
4. If the protocol supplies a signed timestamp, enforce a bounded age and record the delivery ID to resist replay. Signature validity alone does not prove freshness.
5. Insert the raw event metadata and payload hash into an inbox table with a unique key, acknowledge quickly, and process asynchronously.

During secret rotation, accept the old and new secret for a short, audited overlap, but keep the accepted key version on the delivery record. Never log bearer tokens, webhook secrets, full sensitive payloads, or unredacted model prompts. A correlation ID is useful; a credential is not.

3. Delivery semantics are end-to-end properties

Brokers describe transport behavior; the business outcome also depends on producers, consumers, databases, and external side effects. Say exactly where duplication or loss can occur.

At-most-once, at-least-once, and the exactly-once boundary

• At-most-once: do not retry after uncertainty. Work may be lost, but duplicates are avoided. Appropriate only when loss is cheaper than duplication or can be repaired elsewhere.
• At-least-once: retry until acknowledged. Duplicates are expected, so handlers must make repeated delivery harmless.
• Effectively once: use idempotency, uniqueness, and atomic state transitions so the observable business result occurs once within a defined boundary.

Kafka supports idempotent production and transactions within its own processing model, but an interview answer should not stretch that guarantee across an arbitrary email, payroll API, and database. Define the transaction boundary and compensation path. The official Kafka delivery-semantics documentation is a useful vocabulary check.

Inbox, outbox, and idempotent effects

BEGIN;
INSERT INTO processed_event(tenant_id, event_id)
VALUES (:tenant, :event)
ON CONFLICT DO NOTHING;             -- duplicate becomes a no-op

-- Continue only if one row was inserted.
UPSERT employee ...;
INSERT INTO outbox(event_id, aggregate_id, event_type, payload) ...;
COMMIT;

The inbox prevents repeated consumption from applying the same state transition. The outbox stores a domain event in the same database transaction as the domain change; a relay publishes committed rows and records attempts. This removes the “database committed, publish crashed” dual-write gap. It does not eliminate duplicate publication, so consumers still deduplicate. AWS’s primary pattern guide makes both points explicit: transactional outbox pattern.

For non-idempotent external effects, use a provider idempotency key if available. Otherwise introduce a local operation record with a unique business key and a state machine, then reconcile uncertain outcomes by querying the provider before retrying. Never assume a client timeout means the remote call failed.

Ordering, backpressure, and dead letters

Global ordering is expensive and rarely required. Partition by the smallest aggregate that needs order—often tenant_id + employee_id—and include an aggregate version. A consumer can reject stale versions, buffer a short gap, or trigger a focused refresh. Backpressure is a correctness control: bound worker concurrency, honor provider rate-limit hints, apply exponential backoff with full jitter, and stop admitting optional backfills before real-time changes starve.

A dead-letter queue is quarantine, not a cemetery. Store error class, schema version, attempt history, first/last failure time, and a redacted payload reference. Provide replay tooling that revalidates authorization and schema, rate-limits release, and is itself audited. Permanent validation errors belong in a review flow; transient dependency failures normally remain on the retry path.

4. Worked system: a resilient HRIS-to-AI knowledge connector

This is a hypothetical system for interview practice, not a claim about Purnendu’s experience. A customer wants employee directory and policy documents synchronized from an HRIS into an access-controlled AI assistant. The assistant may be minutes stale, but must not expose one tenant’s or one employee group’s data to another.

Architecture and state

Key records are connection (tenant, provider, scopes, encrypted credential reference, token version, health), sync_job (cursor, high-water mark, state, counts), inbox_delivery, source_object (remote ID, source version, payload hash, tombstone), employee, outbox_event, and sync_anomaly. Every tenant-owned key begins with tenant_id; repository methods require tenant context rather than accepting an optional filter.

The incremental sync captures a start high-water mark, pages deterministically, and advances the committed checkpoint only after each page’s effects are durable. Webhooks reduce latency but are hints, not the sole truth. A periodic reconciliation compares provider IDs, counts by status, and sampled hashes; it discovers missed webhooks, silent permission loss, and drift. Deletions become tombstones and downstream removal events, with retention determined by policy.

Failure table

Python backend judgment

Use async def for libraries that expose genuinely asynchronous network or database operations, and offload CPU-heavy parsing or blocking SDKs rather than calling them on the event loop. FastAPI’s own guidance distinguishes waiting on I/O from CPU parallelism: concurrency and async/await. Bound concurrency with a semaphore, pass a deadline through calls, close clients in application lifespan hooks, and treat cancellation as part of control flow rather than a generic error.

async with asyncio.timeout(job.remaining_seconds()):
    async with provider_slots:             # protects provider and this process
        page = await client.list_people(cursor=job.cursor)
    await repository.apply_page_atomically(page, job_id=job.id)
    await repository.commit_checkpoint(job.id, page.next_cursor)

# Important: do not catch BaseException and swallow CancelledError.

Validate external payloads into strict adapter models, then map them to domain types. Unit-test pure mappings; integration-test database transactions and queue redelivery; contract-test representative provider fixtures; and run end-to-end tests against a fake provider that can inject 429s, timeouts, malformed pages, cursor loops, and duplicate webhooks. In a secondary stack, look for the same semantics: Node promises still need cancellation/timeouts and bounded concurrency; Go contexts still need propagation and prompt cancellation.

Syllabus checkpoint: complete connector and backend breadth

API surface and traffic controls

REST and GraphQL both need authentication, tenant scoping, pagination, stable error contracts, observability, and abuse protection. Rate limits should expose a documented budget and retry hint; clients should combine deadlines, exponential backoff, jitter, and circuit breakers without retrying authorization or validation failures. For API versioning, prefer additive evolution, explicit deprecation windows, contract tests, and telemetry proving old versions are no longer used before removal.

Queues are products with different delivery contracts

Kafka is a partitioned log with replay and consumer offsets; SQS is a managed queue with standard/FIFO choices and visibility timeouts; Google Pub/Sub uses acknowledged delivery and subscription-specific retention. Do not flatten them into “a message broker.” State ordering scope, duplication behavior, retention/replay, backpressure, dead-letter policy, maximum payload, and the idempotent side-effect boundary. An RBAC model should restrict who can publish, consume, replay, inspect payloads, and administer schemas per tenant or environment.

Python service depth

FastAPI dependency injection can provide verified principals, tenant context, database sessions, and policy objects, but keep side effects explicit and cleanup deterministic. Use background jobs only for best-effort work that may be lost with the process; durable business work belongs in a queue with status and recovery. Pytest fixtures should establish isolated state, while unit, integration, contract, duplicate-delivery, timeout, and cancellation tests cover different boundaries.

Secondary-stack fluency

For Node.js and TypeScript, understand the event loop, promises/cancellation conventions, type narrowing, runtime validation, and package risk. In Go, understand contexts, goroutines, channels, errors, interfaces, and resource ownership. React breadth should cover state, async server interaction, accessible loading/error states, and streaming UI safety. GraphQL needs resolver authorization and N+1 awareness. PHP matters only for a role that genuinely uses it, such as parts of the Automattic ecosystem. The goal is production-strong Python plus the ability to read, implement, and critically review one secondary stack—not to claim equal depth everywhere.

Interview playbook

Lead with the business invariant, then trace one request and one failure. A compact answer structure is BOUNDARY:

1. B — Business truth: source of truth, freshness, deletion, conflicts, and success metric.
2. O — Ownership and identity: tenant, principal, scopes, data classification, and audit actor.
3. U — Uncertainty: timeouts, duplicates, partial failure, ordering, and unknown outcomes.
4. N — Normalized contract: canonical model, API/job state, event envelope, and versioning.
5. D — Durability: inbox/outbox, checkpoints, idempotency boundary, and reconciliation.
6. A — Admission control: limits, backpressure, deadlines, retry budgets, and circuit breakers.
7. R — Recovery and rollout: replay, DLQ, credential rotation, canary tenants, rollback.
8. Y — Yardsticks: sync lag, completion rate, reconciliation drift, duplicates suppressed, and tenant-scoped errors.

Common traps are saying “exactly once” without a boundary, using email as an immutable identity, trusting a tenant ID from an unsigned payload, retrying every error, treating a DLQ as recovery, or drawing Kafka before clarifying scale. When coding, narrate cancellation, transaction scope, resource cleanup, and how the test proves duplicate safety.

Question bank

Practise these aloud. Each answer should state assumptions and defend one concrete boundary.

Proof artifact: resilient connector laboratory

Build a small, provider-neutral employee connector. All target numbers below are example acceptance thresholds, not claims of prior results.

1. Create a FastAPI control API for connection setup, sync-job creation, status, cancellation, and webhook intake. Use strict request/response models and a machine-readable error envelope.
2. Implement PostgreSQL tables for connection, job/checkpoint, inbox, canonical employee, outbox, and anomalies. Add tenant-scoped unique constraints.
3. Build a fake HRIS with cursor pagination, OAuth token expiry, configurable 429/500/timeout responses, mutable records, deletions, and signed duplicate webhooks.
4. Run a bounded worker that resumes from checkpoints, uses jittered retries and deadlines, atomically applies pages, and relays the outbox to a fake index.
5. Add reconciliation that reports source-only, destination-only, version, and payload-hash mismatches, then offers an audited targeted repair.

Measure: job completion and partial-failure rate, p50/p95 sync lag, queue age, provider calls per changed record, retry count by class, duplicate deliveries suppressed, reconciliation drift, DLQ age, and cross-tenant access test results. Example goals might be zero duplicate business effects in 1,000 repeated deliveries and checkpointed recovery without a full restart; label them as lab targets.

Inject failures: kill the worker after database commit but before acknowledgement; replay one webhook 100 times; expire a token while two workers refresh; return the same cursor twice; add an unknown enum; throttle one tenant; make indexing time out after applying the write; and disconnect a tenant during a backfill. Record expected, observed, and repaired state.

Present: a two-minute architecture walkthrough, state-machine and schema diagram, one trace across webhook-to-index, before/after reconciliation report, automated failure-test output, and a short decision record covering at-least-once delivery, tenant isolation, and one rejected alternative.

Chapter review

A production connector assumes change, duplication, delay, partial failure, and revoked authority. Its core is not the HTTP client: it is an explicit contract plus durable state, idempotent effects, bounded work, reconciliation, and auditable tenant-aware recovery.

Glossary

Canonical model

A stable internal representation that isolates the domain from provider-specific schemas.

High-water mark

A durable boundary showing how far an incremental process has safely advanced.

Idempotency

The property that repeating an operation with the same identity has no additional business effect.

Inbox / outbox

Durable tables that deduplicate received messages and atomically stage messages to publish.

Reconciliation

Comparison of source and destination truth to detect and repair drift that event delivery missed.

Retry budget

A bound on extra attempts so recovery traffic cannot overwhelm a degraded dependency.

Saga

A durable multi-step business process using local transactions and semantic compensation.

Tombstone

An explicit marker that a source object was deleted, enabling downstream removal and audit.

Mastery checklist

• I can define an idempotency key’s scope, retention, concurrency behavior, and stored result.
• I can trace tenant identity from authenticated ingress through queue, database, cache, and audit log.
• I can explain why outbox publication and end-to-end exactly-once effects are different claims.
• I can recover missed webhooks using checkpoints, overlap, replay, and reconciliation.
• I can classify errors into terminal, retryable, uncertain, and human-repair states.
• I can implement bounded async I/O and preserve cancellation and cleanup.
• I can demonstrate duplicate, outage, schema-change, and tenant-isolation tests.