50 / 60 · Transactions

Online Auction — Production Architecture

All designs

Run timed auctions, enforce bid ordering, extend deadlines, and settle winners consistently.

1M concurrent bidders · p99 bid < 150 ms · hot celebrity lots
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT ONLINE AUCTION · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 EXTERNAL Bidder Client interaction boundary 02 EDGE Auction Gateway protocol · auth · quota 03 SERVICE Identity/Deposit domain subsystem 04 STREAM Auction Catalog durable ordered events 05 EDGE Auction Router routing & ownership 06 SERVICE Bid Sequencer domain subsystem 07 SERVICE Auction StateMachine domain subsystem 08 STREAM Bid Journal durable ordered events 09 SERVICE Highest-BidProjection domain subsystem 10 SERVICE Proxy Bid Engine decision & coordination 11 SERVICE Close Timer domain subsystem 12 SERVICE Payment/Settlement domain subsystem 13 STREAM Notification Stream durable ordered events 14 SECURITY Fraud/Collusion domain subsystem 15 OPS Audit/Replay evidence & operations externaledgeservicedatastreamsecurityops
Step 1 / 8Bidder Client

The caller starts the operation at Bidder Client with a stable request and retry identity.

  1. 01 · Bidder ClientThe caller starts the operation at Bidder Client with a stable request and retry identity.
  2. 02 · Auction GatewayAuthenticate bidder/deposit and route all commands for an auction to one sequencer.
  3. 03 · Identity / DepositValidate state, increment, reserve, and dedupe client bid before journal append.
  4. 04 · Fraud / CollusionRun deterministic proxy-bid comparison and update highest bid/close extension atomically.
  5. 05 · Auction RouterPublish bid events and schedule versioned close timer against server time.
  6. 06 · Bid SequencerAt close freeze winner, start settlement, and preserve journal for disputes/fraud.
  7. 07 · Auction CatalogAuction Catalog durably records side effects so workers can replay them independently.
  8. 08 · Audit / ReplayAudit / Replay observes completion and proves the business invariant after commit.
2. Critical request and recovery sequence
MUTATION / COMMITBidder ClientACTOR 01AuctionGatewayACTOR 02AuctionCatalogACTOR 03Bid JournalACTOR 04ONLINE / READ + RECOVERYBidder ClientACTOR 01Identity/DepositACTOR 02Bid SequencerACTOR 03Audit/ReplayACTOR 04Authenticate bidder/deposit and / route all commands for an auctionValidate state, increment, reserve, / and dedupe client bid beforeRun deterministic proxy-bid / comparison and update highestcommitted result + eventRun deterministic proxy-bid / comparison and update highestPublish bid events and schedule / versioned close timer againstAt close freeze winner, start / settlement, and preserve journalcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

Bidder Client

  • Owns client protocol, request identity, retry, and resume behavior.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Auction Gateway

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Identity / Deposit

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Auction Catalog

  • Preserves durable ordering, replay, and consumer checkpoints.
  • Append-only offsets and independently checkpointed consumers.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary online auction workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for 1M concurrent bidders · p99 bid < 150 ms · hot celebrity lots.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: no scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 1M concurrent bidders · p99 bid < 150 ms · hot celebrity lots. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
Traffic1M concurrent bidders · p99 bid < 150 ms · hot celebrity lotsSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

POST /v1/auctions/{id}/bids
GET /v1/auctions/{id}?after_sequence=
POST /v1/auctions/{id}/proxy-bids
POST /v1/auctions/{id}/closeinternal

Authoritative records and indexes

auction(id PK, state, close_at, version, reserve)

bid(auction_id, sequence, bidder, amount, client_bid_id)

proxy_bid(auction_id, bidder, max_amount, version)

settlement(auction_id, winner, amount, state)

Command and event envelopes

# Command envelope
operation: POST /v1/auctions/{id}/bids
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: auction-platform_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: auction-platform_123
event_type: auction_platform_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
Bidder ClientOwns client protocol, request identity, retry, and resume behavior. In this design it participates in: Authenticate bidder/deposit and route all commands for an auction to one sequencer.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Auction Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two bids arrive at same timestamp” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Bidder Client boundary before extracting or scaling it.
Auction GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Validate state, increment, reserve, and dedupe client bid before journal append.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Bidder Client from also owning the contract and failure behavior of Identity / Deposit.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Close timer fires before last committed bid” to Bidder Client.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Auction Gateway boundary before extracting or scaling it.
Identity / DepositOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Run deterministic proxy-bid comparison and update highest bid/close extension atomically.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auction Gateway from also owning the contract and failure behavior of Auction Catalog.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Hot auction exceeds one sequencer” to Auction Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Identity / Deposit boundary before extracting or scaling it.
Auction CatalogPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Publish bid events and schedule versioned close timer against server time.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Identity / Deposit from also owning the contract and failure behavior of Auction Router.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Winning payment fails” to Identity / Deposit.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Auction Catalog boundary before extracting or scaling it.
Auction RouterOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At close freeze winner, start settlement, and preserve journal for disputes/fraud.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auction Catalog from also owning the contract and failure behavior of Bid Sequencer.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Proxy bids reveal private maximum” to Auction Catalog.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Auction Router boundary before extracting or scaling it.
Bid SequencerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate bidder/deposit and route all commands for an auction to one sequencer.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auction Router from also owning the contract and failure behavior of Auction State Machine.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two bids arrive at same timestamp” to Auction Router.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Bid Sequencer boundary before extracting or scaling it.
Auction State MachineOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Validate state, increment, reserve, and dedupe client bid before journal append.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Bid Sequencer from also owning the contract and failure behavior of Bid Journal.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Close timer fires before last committed bid” to Bid Sequencer.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Auction State Machine boundary before extracting or scaling it.
Bid JournalPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Run deterministic proxy-bid comparison and update highest bid/close extension atomically.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auction State Machine from also owning the contract and failure behavior of Highest-Bid Projection.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Hot auction exceeds one sequencer” to Auction State Machine.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Bid Journal boundary before extracting or scaling it.
Highest-Bid ProjectionOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Publish bid events and schedule versioned close timer against server time.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Bid Journal from also owning the contract and failure behavior of Proxy Bid Engine.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Winning payment fails” to Bid Journal.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Highest-Bid Projection boundary before extracting or scaling it.
Proxy Bid EngineOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At close freeze winner, start settlement, and preserve journal for disputes/fraud.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Highest-Bid Projection from also owning the contract and failure behavior of Close Timer.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Proxy bids reveal private maximum” to Highest-Bid Projection.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Proxy Bid Engine boundary before extracting or scaling it.
Close TimerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate bidder/deposit and route all commands for an auction to one sequencer.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Proxy Bid Engine from also owning the contract and failure behavior of Payment / Settlement.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Two bids arrive at same timestamp” to Proxy Bid Engine.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Close Timer boundary before extracting or scaling it.
Payment / SettlementOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Validate state, increment, reserve, and dedupe client bid before journal append.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Close Timer from also owning the contract and failure behavior of Notification Stream.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Close timer fires before last committed bid” to Close Timer.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Payment / Settlement boundary before extracting or scaling it.
Notification StreamPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Run deterministic proxy-bid comparison and update highest bid/close extension atomically.Decouples bursts and failure from user acknowledgment. It prevents Payment / Settlement from also owning the contract and failure behavior of Fraud / Collusion.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Hot auction exceeds one sequencer” to Payment / Settlement.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Notification Stream boundary before extracting or scaling it.
Fraud / CollusionOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Publish bid events and schedule versioned close timer against server time.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Notification Stream from also owning the contract and failure behavior of Audit / Replay.Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Winning payment fails” to Notification Stream.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Fraud / Collusion boundary before extracting or scaling it.
Audit / ReplayOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At close freeze winner, start settlement, and preserve journal for disputes/fraud.Proves correctness and supports controlled recovery. It prevents Fraud / Collusion from also owning the contract and failure behavior of operators and repair tooling.Infer correctness from request logs. Logs do not prove durable business state, detect silent drift, or safely repair ambiguous outcomes. Here that simpler choice would couple “Proxy bids reveal private maximum” to Fraud / Collusion.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Audit / Replay boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Serialize bids per auction partition to enforce monotonic price and deterministic tie ordering; cross-auction work scales independently.

A single deployable with one transactional store is the right starting point for a small online auction workload. The proposed boundary becomes justified when serialize bids per auction partition to enforce monotonic price and deterministic tie ordering; cross-auction work scales independently. Separating it contains the named failure “Two bids arrive at same timestamp” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

A bid journal is authoritative evidence, while current highest bid is a projection guarded by expected sequence.

The design coordinates only where no scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Use server time and a versioned closing transition; anti-sniping extension must be part of the atomic bid decision, not a later timer.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Run deterministic proxy-bid comparison and update highest bid/close extension atomically.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect no scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Serialize bids per auction partition to enforce monotonic price and deterministic tie ordering; cross-auction work scales independently.

Scale boundary

A bid journal is authoritative evidence, while current highest bid is a projection guarded by expected sequence.

Operational boundary

Use server time and a versioned closing transition; anti-sniping extension must be part of the atomic bid decision, not a later timer.

↑ top

5. Critical flow and algorithm

01

Authenticate bidder/deposit and route all commands for an auction to one sequencer.

02

Validate state, increment, reserve, and dedupe client bid before journal append.

03

Run deterministic proxy-bid comparison and update highest bid/close extension atomically.

04

Publish bid events and schedule versioned close timer against server time.

05

At close freeze winner, start settlement, and preserve journal for disputes/fraud.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Add a cache only for a measured repeated read; do not place mutable correctness state in a disposable layer.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

no scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Two bids arrive at same timestampResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Close timer fires before last committed bidStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Hot auction exceeds one sequencerIsolate the key/tenant/priority, apply admission control, and preserve durable work.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Winning payment failsStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Proxy bids reveal private maximumStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Two bids arrive at same timestamp
operation_id: op_01J...
aggregate_key: auction-platform_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Audit / Replay

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Two bids arrive at same timestamp — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

How do you handle simultaneous bids?

Serialize bids per auction partition to enforce monotonic price and deterministic tie ordering; cross-auction work scales independently. Walk the concrete lifecycle step “Authenticate bidder/deposit and route all commands for an auction to one sequencer.” Then test it against “Two bids arrive at same timestamp”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How does anti-sniping work?

A bid journal is authoritative evidence, while current highest bid is a projection guarded by expected sequence. Walk the concrete lifecycle step “Validate state, increment, reserve, and dedupe client bid before journal append.” Then test it against “Close timer fires before last committed bid”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How are payment failures handled?

Use server time and a versioned closing transition; anti-sniping extension must be part of the atomic bid decision, not a later timer. Walk the concrete lifecycle step “Run deterministic proxy-bid comparison and update highest bid/close extension atomically.” Then test it against “Hot auction exceeds one sequencer”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do proxy bids work?

Serialize bids per auction partition to enforce monotonic price and deterministic tie ordering; cross-auction work scales independently. Walk the concrete lifecycle step “Publish bid events and schedule versioned close timer against server time.” Then test it against “Winning payment fails”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you shard a hot auction?

A bid journal is authoritative evidence, while current highest bid is a projection guarded by expected sequence. Walk the concrete lifecycle step “At close freeze winner, start settlement, and preserve journal for disputes/fraud.” Then test it against “Proxy bids reveal private maximum”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invariantno scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects no scarce resource or monetary effect is created twice, lost after acknowledgment, or committed without an audit trail. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top