46 / 60 · Geo & Realtime

Food Delivery Marketplace — Production Architecture

All designs

Coordinate menus, orders, restaurants, couriers, dispatch, and settlement in real time.

5M orders/day · courier updates every 5 s · dispatch < 3 s
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT FOOD DELIVERY MARKETPLACE · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 EXTERNAL Customer App interaction boundary 02 EXTERNAL Restaurant App interaction boundary 03 EXTERNAL Courier App interaction boundary 04 EDGE Realtime Gateway protocol · auth · quota 05 SERVICE Menu/Availability domain subsystem 06 SERVICE Order API protocol · auth · quota 07 SERVICE Pricing/Fees domain subsystem 08 SERVICE Payment Intent domain subsystem 09 STREAM Restaurant TabletQueue durable ordered events 10 SERVICE Preparation Tracker domain subsystem 11 SERVICE Courier Geo Index low-latency read path 12 SERVICE Dispatch Engine decision & coordination 13 DATA Offer Lease Store authoritative durable state 14 SERVICE Delivery StateMachine domain subsystem 15 SERVICE Settlement/Support domain subsystem externaledgeservicedatastreamsecurityops
Step 1 / 6Customer App

The caller starts the operation at Customer App with a stable request and retry identity.

  1. 01 · Customer AppThe caller starts the operation at Customer App with a stable request and retry identity.
  2. 02 · Realtime GatewayQuote menu, fees, tax, and payment intent under expiring versions.
  3. 03 · Menu / AvailabilityCommit order and ask restaurant to accept before charging/capturing per policy.
  4. 04 · Dispatch EngineTrack preparation and search geo cells for eligible courier candidates.
  5. 05 · Offer Lease StoreLease one courier offer conditionally and advance pickup/delivery state by version.
  6. 06 · Restaurant Tablet QueueSettle restaurant/courier and compensate cancellations from the durable event trail.
2. Critical request and recovery sequence
MUTATION / COMMITCustomer AppACTOR 01Restaurant AppACTOR 02RealtimeGatewayACTOR 03Payment IntentACTOR 04ONLINE / READ + RECOVERYCustomer AppACTOR 01Courier AppACTOR 02Order APIACTOR 03Settlement/SupportACTOR 04Quote menu, fees, tax, and payment / intent under expiring versions.Commit order and ask restaurant to / accept before charging/capturingTrack preparation and search geo / cells for eligible couriercommitted result + eventTrack preparation and search geo / cells for eligible courierLease one courier offer / conditionally and advanceSettle restaurant/courier and / compensate cancellations from thecached/read result + cursortimeout → retry same identity → read committed truth → reconcile

Customer App

  • Owns client protocol, request identity, retry, and resume behavior.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Restaurant App

  • Owns client protocol, request identity, retry, and resume behavior.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Courier App

  • Owns client protocol, request identity, retry, and resume behavior.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Realtime Gateway

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary food delivery marketplace workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for 5M orders/day · courier updates every 5 s · dispatch < 3 s.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: each accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 5M orders/day · courier updates every 5 s · dispatch < 3 s. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
Traffic5M orders/day · courier updates every 5 s · dispatch < 3 sSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

GET /v1/restaurants?location=available catalog
POST /v1/ordersidempotent order
POST /v1/orders/{id}/transitionsstate move
POST /v1/couriers/{id}/locationsbatched GPS

Authoritative records and indexes

order(order_id PK, customer, restaurant, state, version)

order_item(order_id, menu_item_version, qty, price)

courier_offer(offer_id, order_id, courier_id, lease_until)

settlement(order_id, restaurant_amount, courier_amount, platform_fee)

Command and event envelopes

# Command envelope
operation: GET /v1/restaurants?location=
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: food-delivery_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: food-delivery_123
event_type: food_delivery_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
Customer AppOwns client protocol, request identity, retry, and resume behavior. In this design it participates in: Quote menu, fees, tax, and payment intent under expiring versions.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Restaurant App.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Restaurant accepts after customer cancellation” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Customer App boundary before extracting or scaling it.
Restaurant AppOwns client protocol, request identity, retry, and resume behavior. In this design it participates in: Commit order and ask restaurant to accept before charging/capturing per policy.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Customer App from also owning the contract and failure behavior of Courier App.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two couriers accept same offer” to Customer App.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Restaurant App boundary before extracting or scaling it.
Courier AppOwns client protocol, request identity, retry, and resume behavior. In this design it participates in: Track preparation and search geo cells for eligible courier candidates.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Restaurant App from also owning the contract and failure behavior of Realtime Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “GPS is stale near pickup” to Restaurant App.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Courier App boundary before extracting or scaling it.
Realtime GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Lease one courier offer conditionally and advance pickup/delivery state by version.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Courier App from also owning the contract and failure behavior of Menu / Availability.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Payment succeeds but order commit times out” to Courier App.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Realtime Gateway boundary before extracting or scaling it.
Menu / AvailabilityOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Settle restaurant/courier and compensate cancellations from the durable event trail.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Realtime Gateway from also owning the contract and failure behavior of Order API.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Restaurant tablet offline creates backlog” to Realtime Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Menu / Availability boundary before extracting or scaling it.
Order APIOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Quote menu, fees, tax, and payment intent under expiring versions.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Menu / Availability from also owning the contract and failure behavior of Pricing / Fees.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Restaurant accepts after customer cancellation” to Menu / Availability.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Order API boundary before extracting or scaling it.
Pricing / FeesOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Commit order and ask restaurant to accept before charging/capturing per policy.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Order API from also owning the contract and failure behavior of Payment Intent.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two couriers accept same offer” to Order API.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Pricing / Fees boundary before extracting or scaling it.
Payment IntentOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Track preparation and search geo cells for eligible courier candidates.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Pricing / Fees from also owning the contract and failure behavior of Restaurant Tablet Queue.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “GPS is stale near pickup” to Pricing / Fees.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Payment Intent boundary before extracting or scaling it.
Restaurant Tablet QueuePreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Lease one courier offer conditionally and advance pickup/delivery state by version.Decouples bursts and failure from user acknowledgment. It prevents Payment Intent from also owning the contract and failure behavior of Preparation Tracker.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Payment succeeds but order commit times out” to Payment Intent.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Restaurant Tablet Queue boundary before extracting or scaling it.
Preparation TrackerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Settle restaurant/courier and compensate cancellations from the durable event trail.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Restaurant Tablet Queue from also owning the contract and failure behavior of Courier Geo Index.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Restaurant tablet offline creates backlog” to Restaurant Tablet Queue.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Preparation Tracker boundary before extracting or scaling it.
Courier Geo IndexServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Quote menu, fees, tax, and payment intent under expiring versions.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Preparation Tracker from also owning the contract and failure behavior of Dispatch Engine.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Restaurant accepts after customer cancellation” to Preparation Tracker.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (5M orders/day · courier updates every 5 s · dispatch < 3 s) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Courier Geo Index boundary before extracting or scaling it.
Dispatch EngineOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Commit order and ask restaurant to accept before charging/capturing per policy.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Courier Geo Index from also owning the contract and failure behavior of Offer Lease Store.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two couriers accept same offer” to Courier Geo Index.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Dispatch Engine boundary before extracting or scaling it.
Offer Lease StoreOwns authoritative versioned state or immutable evidence. In this design it participates in: Track preparation and search geo cells for eligible courier candidates.Defines the source of truth and commit boundary. It prevents Dispatch Engine from also owning the contract and failure behavior of Delivery State Machine.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “GPS is stale near pickup” to Dispatch Engine.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Offer Lease Store boundary before extracting or scaling it.
Delivery State MachineOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Lease one courier offer conditionally and advance pickup/delivery state by version.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Offer Lease Store from also owning the contract and failure behavior of Settlement / Support.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Payment succeeds but order commit times out” to Offer Lease Store.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Delivery State Machine boundary before extracting or scaling it.
Settlement / SupportOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Settle restaurant/courier and compensate cancellations from the durable event trail.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Delivery State Machine from also owning the contract and failure behavior of operators and repair tooling.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Restaurant tablet offline creates backlog” to Delivery State Machine.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Settlement / Support boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

The order is a versioned state machine; restaurant acceptance, payment, preparation, pickup, and delivery are not one distributed transaction.

A single deployable with one transactional store is the right starting point for a small food delivery marketplace workload. The proposed boundary becomes justified when the order is a versioned state machine; restaurant acceptance, payment, preparation, pickup, and delivery are not one distributed transaction. Separating it contains the named failure “Restaurant accepts after customer cancellation” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Live courier location belongs in an ephemeral geo index while order, assignment, and settlement remain durable.

The design coordinates only where each accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Dispatch optimizes ETA and fairness with short offers/leases; forcing one courier choice synchronously makes stale GPS and disconnects catastrophic.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Track preparation and search geo cells for eligible courier candidates.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect each accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

The order is a versioned state machine; restaurant acceptance, payment, preparation, pickup, and delivery are not one distributed transaction.

Scale boundary

Live courier location belongs in an ephemeral geo index while order, assignment, and settlement remain durable.

Operational boundary

Dispatch optimizes ETA and fairness with short offers/leases; forcing one courier choice synchronously makes stale GPS and disconnects catastrophic.

↑ top

5. Critical flow and algorithm

01

Quote menu, fees, tax, and payment intent under expiring versions.

02

Commit order and ask restaurant to accept before charging/capturing per policy.

03

Track preparation and search geo cells for eligible courier candidates.

04

Lease one courier offer conditionally and advance pickup/delivery state by version.

05

Settle restaurant/courier and compensate cancellations from the durable event trail.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

each accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Restaurant accepts after customer cancellationStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Two couriers accept same offerResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
GPS is stale near pickupStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Payment succeeds but order commit times outStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Restaurant tablet offline creates backlogIsolate the key/tenant/priority, apply admission control, and preserve durable work.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Restaurant accepts after customer cancellation
operation_id: op_01J...
aggregate_key: food-delivery_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Settlement / Support

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Restaurant accepts after customer cancellation — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

How do restaurants update availability?

The order is a versioned state machine; restaurant acceptance, payment, preparation, pickup, and delivery are not one distributed transaction. Walk the concrete lifecycle step “Quote menu, fees, tax, and payment intent under expiring versions.” Then test it against “Restaurant accepts after customer cancellation”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you avoid two couriers accepting?

Live courier location belongs in an ephemeral geo index while order, assignment, and settlement remain durable. Walk the concrete lifecycle step “Commit order and ask restaurant to accept before charging/capturing per policy.” Then test it against “Two couriers accept same offer”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How is ETA predicted?

Dispatch optimizes ETA and fairness with short offers/leases; forcing one courier choice synchronously makes stale GPS and disconnects catastrophic. Walk the concrete lifecycle step “Track preparation and search geo cells for eligible courier candidates.” Then test it against “GPS is stale near pickup”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do cancellations compensate?

The order is a versioned state machine; restaurant acceptance, payment, preparation, pickup, and delivery are not one distributed transaction. Walk the concrete lifecycle step “Lease one courier offer conditionally and advance pickup/delivery state by version.” Then test it against “Payment succeeds but order commit times out”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How are batched deliveries handled?

Live courier location belongs in an ephemeral geo index while order, assignment, and settlement remain durable. Walk the concrete lifecycle step “Settle restaurant/courier and compensate cancellations from the durable event trail.” Then test it against “Restaurant tablet offline creates backlog”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invarianteach accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects each accepted food delivery marketplace operation has one stable result and retries do not repeat its business effect. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top