API Gateway — System Design Academy

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

Execute the primary api gateway workflow and return an observable result.
Expose the concrete operations shown below with stable retry identities.
Allow operators to inspect, disable, repair, or replay ambiguous state.
Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

Design for 2M requests/s · p99 overhead < 10 ms · global edge.
Protect the named business invariant under concurrency and partial failure.
Scale by an explicit aggregate/partition key without one global mutex.
Degrade intentionally when cache, queue, store, or regional ownership is unavailable.

Primary invariant: each accepted api gateway operation has one stable result and retries do not repeat its business effect.

↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 2M requests/s · p99 overhead < 10 ms · global edge. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

Dimension	Working estimate	Design implication
Traffic	2M requests/s · p99 overhead < 10 ms · global edge	Separate online compute, durable state, and asynchronous consumers.
Peak factor	5–10× average	Admission control, autoscaling headroom, and hot-key protection.
Stored state	write rate × retention × record bytes	Partition count, indexes, compaction, backup, and archival tier.
Network	peak operations × payload size	Batching, compression, edge placement, and egress budget.
Availability	99.9–99.99%	Multi-zone replicas, tested failover, and dependency error budgets.
Recovery	RPO/RTO by invariant	Ack boundary, replay retention, reconciliation, and operator runbooks.

Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.

↑ top

3. API and data model

PUT /v1/routes/{id}versioned route config

PUT /v1/policies/{id}auth/limit/retry policy

GET /v1/snapshots/{version}signed data-plane bundle

PROXY /{path}deadline-bound request

Authoritative records and indexes

route(route_id, host, path, methods, cluster, policy_refs, version)

cluster(cluster_id, endpoints, health_policy, protocol)

snapshot(version PK, checksum, signature, created_at)

request_log(trace_id, route_id, outcome, latency_bucket)

Command and event envelopes

# Command envelope
operation: PUT /v1/routes/{id}
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: api-gateway_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: api-gateway_123
event_type: api_gateway_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

Component	Owned responsibility	Why separate it	Rejected alternative	Cost and evolution trigger
API Client	Owns client protocol, request identity, retry, and resume behavior. In this design it participates in: Terminate TLS, reject malformed traffic, and propagate a request deadline.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Anycast / DNS.	Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Bad snapshot reaches part of fleet” to the external caller.	Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the API Client boundary before extracting or scaling it.
Anycast / DNS	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate, authorize the route/resource, and apply local or distributed quotas.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents API Client from also owning the contract and failure behavior of DDoS / WAF.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Retry storm amplifies backend outage” to API Client.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Anycast / DNS boundary before extracting or scaling it.
DDoS / WAF	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Match a route from one immutable snapshot and select a healthy backend endpoint.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Anycast / DNS from also owning the contract and failure behavior of TLS Terminator.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Discovery returns zero healthy endpoints” to Anycast / DNS.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the DDoS / WAF boundary before extracting or scaling it.
TLS Terminator	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Retry only safe requests within the remaining budget; trip per-cluster circuits.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents DDoS / WAF from also owning the contract and failure behavior of Gateway Worker.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Large body exhausts gateway memory” to DDoS / WAF.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the TLS Terminator boundary before extracting or scaling it.
Gateway Worker	Owns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Stream bounded telemetry while the data plane continues on last-known-good config.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents TLS Terminator from also owning the contract and failure behavior of JWT / mTLS Auth.	Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Auth provider is unavailable” to TLS Terminator.	Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Gateway Worker boundary before extracting or scaling it.
JWT / mTLS Auth	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Terminate TLS, reject malformed traffic, and propagate a request deadline.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Gateway Worker from also owning the contract and failure behavior of Rate Limiter.	Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Bad snapshot reaches part of fleet” to Gateway Worker.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the JWT / mTLS Auth boundary before extracting or scaling it.
Rate Limiter	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate, authorize the route/resource, and apply local or distributed quotas.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents JWT / mTLS Auth from also owning the contract and failure behavior of Route Matcher.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Retry storm amplifies backend outage” to JWT / mTLS Auth.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Rate Limiter boundary before extracting or scaling it.
Route Matcher	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Match a route from one immutable snapshot and select a healthy backend endpoint.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Rate Limiter from also owning the contract and failure behavior of Service Discovery.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Discovery returns zero healthy endpoints” to Rate Limiter.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Route Matcher boundary before extracting or scaling it.
Service Discovery	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Retry only safe requests within the remaining budget; trip per-cluster circuits.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Route Matcher from also owning the contract and failure behavior of Load Balancer.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Large body exhausts gateway memory” to Route Matcher.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Service Discovery boundary before extracting or scaling it.
Load Balancer	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Stream bounded telemetry while the data plane continues on last-known-good config.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Service Discovery from also owning the contract and failure behavior of Backend Service.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Auth provider is unavailable” to Service Discovery.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Load Balancer boundary before extracting or scaling it.
Backend Service	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Terminate TLS, reject malformed traffic, and propagate a request deadline.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Load Balancer from also owning the contract and failure behavior of Response Cache.	Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Bad snapshot reaches part of fleet” to Load Balancer.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Backend Service boundary before extracting or scaling it.
Response Cache	Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: Authenticate, authorize the route/resource, and apply local or distributed quotas.	Removes repeated work from the latency-critical path. It prevents Backend Service from also owning the contract and failure behavior of Policy Control Plane.	Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Retry storm amplifies backend outage” to Backend Service.	Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (2M requests/s · p99 overhead < 10 ms · global edge) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Response Cache boundary before extracting or scaling it.
Policy Control Plane	Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Match a route from one immutable snapshot and select a healthy backend endpoint.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Response Cache from also owning the contract and failure behavior of Telemetry Pipeline.	Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Discovery returns zero healthy endpoints” to Response Cache.	Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Policy Control Plane boundary before extracting or scaling it.
Telemetry Pipeline	Executes idempotent asynchronous work under lease and retry policy. In this design it participates in: Retry only safe requests within the remaining budget; trip per-cluster circuits.	Separates a distinct scale, ownership, policy, or failure boundary. It prevents Policy Control Plane from also owning the contract and failure behavior of operators and repair tooling.	Perform the work before acknowledging the caller. This gives simpler consistency but unbounded tail latency and fragile retries. Here that simpler choice would couple “Large body exhausts gateway memory” to Policy Control Plane.	Costs leases, checkpoints, retries, DLQs, and lag operations. Separate once execution time or burstiness no longer fits the request deadline. Measure the latency, saturation, lag, or mismatch at the Telemetry Pipeline boundary before extracting or scaling it.

Decision 01

Chosen boundary vs simpler monolith

Keep the data plane stateless and fast; distribute signed, versioned policy snapshots from a control plane.

A single deployable with one transactional store is the right starting point for a small api gateway workload. The proposed boundary becomes justified when keep the data plane stateless and fast; distribute signed, versioned policy snapshots from a control plane. Separating it contains the named failure “Bad snapshot reaches part of fleet” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Apply bounded retries only to idempotent operations and combine them with timeouts and circuit breakers.

The design coordinates only where each accepted api gateway operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Propagate request IDs, identity, budgets, and trace context; reject malformed requests at the edge.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Match a route from one immutable snapshot and select a healthy backend endpoint.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect each accepted api gateway operation has one stable result and retries do not repeat its business effect. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.

Correctness boundary

Keep the data plane stateless and fast; distribute signed, versioned policy snapshots from a control plane.

Scale boundary

Apply bounded retries only to idempotent operations and combine them with timeouts and circuit breakers.

Operational boundary

Propagate request IDs, identity, budgets, and trace context; reject malformed requests at the edge.

↑ top

5. Critical flow and algorithm

Terminate TLS, reject malformed traffic, and propagate a request deadline.

Authenticate, authorize the route/resource, and apply local or distributed quotas.

Match a route from one immutable snapshot and select a healthy backend endpoint.

Retry only safe requests within the remaining budget; trip per-cluster circuits.

Stream bounded telemetry while the data plane continues on last-known-good config.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result

Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.

↑ top

6. Caching, hot keys, and backpressure

Cache policy

Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.

↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

each accepted api gateway operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

Operation	Consistency	Reason
Authoritative mutation	Conditional / serializable	Protect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own write	Primary, session token, or cache warm	The caller must observe the result it just committed.
Ordinary read projection	Bounded stale where safe	Replicas and caches improve latency when version/freshness is visible.
Analytics and operations	Eventual + replay	Freshness is less important than isolating the online path.

Route every mutation for one aggregate to one owner/partition epoch.
Use cursors rather than offsets for changing ordered collections.
Retain tombstones until replication and repair windows close.
Move shard ownership through versioned maps and dual-read verification.

↑ top

8. Failure drills and recovery

Failure	Runtime behavior	Recovery proof
Bad snapshot reaches part of fleet	Isolate the key/tenant/priority, apply admission control, and preserve durable work.	Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Retry storm amplifies backend outage	Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.	Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Discovery returns zero healthy endpoints	Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.	Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Large body exhausts gateway memory	Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.	Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Auth provider is unavailable	Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.	Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Bad snapshot reaches part of fleet
operation_id: op_01J...
aggregate_key: api-gateway_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Telemetry Pipeline

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

Authenticate at ingress and authorize the exact resource/tenant on every operation.
Encrypt transport and durable data; tokenize or isolate sensitive material.
Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
Redact secrets and personal data before logs or analytical storage.
Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

User-visible success rate and p50/p95/p99 latency.
Timeouts, circuit state, saturation, and replica health by dependency.
Queue lag, oldest-item age, retries, and dead-letter growth.
Conflict, duplicate-suppression, and reconciliation mismatch counts.
Business drill: Bad snapshot reaches part of fleet — alert before the user-visible invariant is violated.

Regional evolution

Begin multi-zone with tested failover, backups, and reconciliation.
Add global stateless ingress and regional read projections.
Keep one write owner per aggregate or partition where coordination is expensive.
Move ownership with epochs/fencing; avoid silent simultaneous writers.
Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.

↑ top

10. Interviewer follow-ups

How do configuration updates propagate?

Keep the data plane stateless and fast; distribute signed, versioned policy snapshots from a control plane. Walk the concrete lifecycle step “Terminate TLS, reject malformed traffic, and propagate a request deadline.” Then test it against “Bad snapshot reaches part of fleet”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you avoid retry storms?

Apply bounded retries only to idempotent operations and combine them with timeouts and circuit breakers. Walk the concrete lifecycle step “Authenticate, authorize the route/resource, and apply local or distributed quotas.” Then test it against “Retry storm amplifies backend outage”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How does service discovery work?

Propagate request IDs, identity, budgets, and trace context; reject malformed requests at the edge. Walk the concrete lifecycle step “Match a route from one immutable snapshot and select a healthy backend endpoint.” Then test it against “Discovery returns zero healthy endpoints”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

What changes for WebSockets?

Keep the data plane stateless and fast; distribute signed, versioned policy snapshots from a control plane. Walk the concrete lifecycle step “Retry only safe requests within the remaining budget; trip per-cluster circuits.” Then test it against “Large body exhausts gateway memory”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you isolate tenants?

Apply bounded retries only to idempotent operations and combine them with timeouts and circuit breakers. Walk the concrete lifecycle step “Stream bounded telemetry while the data plane continues on last-known-good config.” Then test it against “Auth provider is unavailable”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.

↑ top

11. How to present this in 45 minutes

Time	What to cover
0–5 min	Clarify scope, workload, SLOs, and the correctness invariant.
5–10 min	Estimate throughput/storage and define concrete APIs and records.
10–20 min	Draw online, durable-write, and asynchronous/repair paths.
20–30 min	Explain the exact lifecycle, partition key, and concurrency control.
30–38 min	Discuss caching, hot keys, replication, and backpressure.
38–43 min	Run failure drills, security, observability, and regional evolution.
43–45 min	Summarize the five most important choices and trade-offs.

Final decision summary

Invariant	each accepted api gateway operation has one stable result and retries do not repeat its business effect.
Partitioning	Stable aggregate/key hash with versioned ownership and measured hot-key isolation.
Commit	Expected version or uniqueness at the authoritative state boundary.
Async work	Durable outbox/log, idempotent consumers, checkpoints, and replay.
Recovery	Same operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects each accepted api gateway operation has one stable result and retries do not repeat its business effect. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”

↑ top

API Client

Anycast / DNS

DDoS / WAF

TLS Terminator

Complete System Design Tutorial

1. Requirements and scope

Functional requirements

Non-functional requirements

2. Capacity assumptions

3. API and data model

Authoritative records and indexes

Command and event envelopes

4. Why this architecture and what it costs

Chosen boundary vs simpler monolith

Correctness vs availability

Operational cost and evolution

5. Critical flow and algorithm

Retry-safe mutation skeleton

6. Caching, hot keys, and backpressure

7. Partitioning and consistency

Strong boundary

Eventual boundary

8. Failure drills and recovery

Recovery checkpoint

9. Security, regions, and observability

Security and abuse

Signals that prove correctness

Regional evolution

10. Interviewer follow-ups

11. How to present this in 45 minutes

Final decision summary

Closing summary