57 / 60 · Distributed Data

Distributed SQL Database — Production Architecture

All designs

Provide SQL transactions over sharded, replicated data with online rebalancing.

PB-scale · millions TPS · serializable option
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT DISTRIBUTED SQL DATABASE · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 EXTERNAL SQL Client interaction boundary 02 EDGE SQL Gateway protocol · auth · quota 03 SERVICE Parser/Optimizer asynchronous execution 04 STREAM Schema Catalog durable ordered events 05 EDGE Range Router routing & ownership 06 SERVICE TransactionCoordinator decision & coordination 07 SERVICE Timestamp Oracle domain subsystem 08 SERVICE Range Leaseholder domain subsystem 09 SERVICE Consensus Replicas domain subsystem 10 SERVICE MVCC Engine decision & coordination 11 DATA Distributed LockTable authoritative durable state 12 STREAM Changefeed Log durable ordered events 13 SERVICE Range Split/Merge domain subsystem 14 SERVICE Backup/Restore authoritative durable state 15 OPS Observability evidence & operations externaledgeservicedatastreamsecurityops
Step 1 / 7SQL Client

The caller starts the operation at SQL Client with a stable request and retry identity.

  1. 01 · SQL ClientThe caller starts the operation at SQL Client with a stable request and retry identity.
  2. 02 · SQL GatewayParse/optimize SQL and route key spans using versioned range descriptors.
  3. 03 · Transaction CoordinatorChoose read timestamp and execute local or distributed reads from MVCC snapshots.
  4. 04 · MVCC EngineFor writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.
  5. 05 · Distributed Lock TableResolve abandoned intents and publish changefeed after commit timestamp is durable.
  6. 06 · Changefeed LogSplit/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.
  7. 07 · ObservabilityObservability observes completion and proves the business invariant after commit.
2. Critical request and recovery sequence
MUTATION / COMMITSQL ClientACTOR 01SQL GatewayACTOR 02Schema CatalogACTOR 03RangeLeaseholderACTOR 04ONLINE / READ + RECOVERYSQL ClientACTOR 01Parser/OptimizerACTOR 02TransactionCoordinatorACTOR 03ObservabilityACTOR 04Parse/optimize SQL and route key / spans using versioned rangeChoose read timestamp and execute / local or distributed reads fromFor writes acquire intents/locks, / replicate each range command bycommitted result + eventFor writes acquire intents/locks, / replicate each range command byResolve abandoned intents and / publish changefeed after commitSplit/merge hot ranges, rebalance / replicas, compact MVCC, and runcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

SQL Client

  • Owns client protocol, request identity, retry, and resume behavior.
  • Stateless or lease-scoped; durable progress lives elsewhere.

SQL Gateway

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Parser / Optimizer

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Schema Catalog

  • Preserves durable ordering, replay, and consumer checkpoints.
  • Append-only offsets and independently checkpointed consumers.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary distributed sql database workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for PB-scale · millions TPS · serializable option.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: acknowledged writes survive the configured failure model and replicas converge.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is PB-scale · millions TPS · serializable option. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
TrafficPB-scale · millions TPS · serializable optionSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

SQL BEGIN/COMMIT transaction
SELECT ... AS OF SYSTEM TIME
ALTER TABLE ... online
WATCH table/range changefeed

Authoritative records and indexes

range(range_id, start_key, end_key, replicas, epoch)

mvcc_cell(key, timestamp, value, tombstone)

transaction(id, read_ts, write_ts, state, participants)

schema(version, descriptor, effective_ts)

Command and event envelopes

# Command envelope
operation: SQL BEGIN/COMMIT transaction
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: distributed-sql_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: distributed-sql_123
event_type: distributed_sql_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
SQL ClientOwns client protocol, request identity, retry, and resume behavior. In this design it participates in: Parse/optimize SQL and route key spans using versioned range descriptors.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of SQL Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Coordinator dies during two-phase commit” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the SQL Client boundary before extracting or scaling it.
SQL GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Choose read timestamp and execute local or distributed reads from MVCC snapshots.Separates a distinct scale, ownership, policy, or failure boundary. It prevents SQL Client from also owning the contract and failure behavior of Parser / Optimizer.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Hot sequential key overloads one range” to SQL Client.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the SQL Gateway boundary before extracting or scaling it.
Parser / OptimizerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.Separates a distinct scale, ownership, policy, or failure boundary. It prevents SQL Gateway from also owning the contract and failure behavior of Schema Catalog.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Long read blocks MVCC garbage collection” to SQL Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Parser / Optimizer boundary before extracting or scaling it.
Schema CatalogPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Resolve abandoned intents and publish changefeed after commit timestamp is durable.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Parser / Optimizer from also owning the contract and failure behavior of Range Router.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Leaseholder moves during transaction” to Parser / Optimizer.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Schema Catalog boundary before extracting or scaling it.
Range RouterOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Split/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Schema Catalog from also owning the contract and failure behavior of Transaction Coordinator.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Schema version changes mid-query” to Schema Catalog.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Range Router boundary before extracting or scaling it.
Transaction CoordinatorOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Parse/optimize SQL and route key spans using versioned range descriptors.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Range Router from also owning the contract and failure behavior of Timestamp Oracle.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Coordinator dies during two-phase commit” to Range Router.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Transaction Coordinator boundary before extracting or scaling it.
Timestamp OracleOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Choose read timestamp and execute local or distributed reads from MVCC snapshots.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Transaction Coordinator from also owning the contract and failure behavior of Range Leaseholder.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Hot sequential key overloads one range” to Transaction Coordinator.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Timestamp Oracle boundary before extracting or scaling it.
Range LeaseholderOwns one bounded domain decision, transition, or operational capability. In this design it participates in: For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Timestamp Oracle from also owning the contract and failure behavior of Consensus Replicas.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Long read blocks MVCC garbage collection” to Timestamp Oracle.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Range Leaseholder boundary before extracting or scaling it.
Consensus ReplicasOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Resolve abandoned intents and publish changefeed after commit timestamp is durable.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Range Leaseholder from also owning the contract and failure behavior of MVCC Engine.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Leaseholder moves during transaction” to Range Leaseholder.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Consensus Replicas boundary before extracting or scaling it.
MVCC EngineOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Split/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Consensus Replicas from also owning the contract and failure behavior of Distributed Lock Table.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Schema version changes mid-query” to Consensus Replicas.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the MVCC Engine boundary before extracting or scaling it.
Distributed Lock TableOwns authoritative versioned state or immutable evidence. In this design it participates in: Parse/optimize SQL and route key spans using versioned range descriptors.Separates a distinct scale, ownership, policy, or failure boundary. It prevents MVCC Engine from also owning the contract and failure behavior of Changefeed Log.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Coordinator dies during two-phase commit” to MVCC Engine.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Distributed Lock Table boundary before extracting or scaling it.
Changefeed LogPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Choose read timestamp and execute local or distributed reads from MVCC snapshots.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Distributed Lock Table from also owning the contract and failure behavior of Range Split / Merge.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Hot sequential key overloads one range” to Distributed Lock Table.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Changefeed Log boundary before extracting or scaling it.
Range Split / MergeOwns one bounded domain decision, transition, or operational capability. In this design it participates in: For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Changefeed Log from also owning the contract and failure behavior of Backup / Restore.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Long read blocks MVCC garbage collection” to Changefeed Log.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Range Split / Merge boundary before extracting or scaling it.
Backup / RestoreOwns authoritative versioned state or immutable evidence. In this design it participates in: Resolve abandoned intents and publish changefeed after commit timestamp is durable.Defines the source of truth and commit boundary. It prevents Range Split / Merge from also owning the contract and failure behavior of Observability.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Leaseholder moves during transaction” to Range Split / Merge.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Backup / Restore boundary before extracting or scaling it.
ObservabilityOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Split/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Backup / Restore from also owning the contract and failure behavior of operators and repair tooling.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Schema version changes mid-query” to Backup / Restore.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Observability boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Range partitions preserve ordered scans while splitting hot ranges; hash sharding is simpler for point workloads but destroys locality.

A single deployable with one transactional store is the right starting point for a small distributed sql database workload. The proposed boundary becomes justified when range partitions preserve ordered scans while splitting hot ranges; hash sharding is simpler for point workloads but destroys locality. Separating it contains the named failure “Coordinator dies during two-phase commit” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Consensus replicates each range; distributed transactions coordinate only the ranges touched and pay latency proportional to geography.

The design coordinates only where acknowledged writes survive the configured failure model and replicas converge. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

MVCC enables consistent snapshots and online reads, but garbage collection must honor active transactions, backups, and change feeds.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect acknowledged writes survive the configured failure model and replicas converge. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Range partitions preserve ordered scans while splitting hot ranges; hash sharding is simpler for point workloads but destroys locality.

Scale boundary

Consensus replicates each range; distributed transactions coordinate only the ranges touched and pay latency proportional to geography.

Operational boundary

MVCC enables consistent snapshots and online reads, but garbage collection must honor active transactions, backups, and change feeds.

↑ top

5. Critical flow and algorithm

01

Parse/optimize SQL and route key spans using versioned range descriptors.

02

Choose read timestamp and execute local or distributed reads from MVCC snapshots.

03

For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.

04

Resolve abandoned intents and publish changefeed after commit timestamp is durable.

05

Split/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Add a cache only for a measured repeated read; do not place mutable correctness state in a disposable layer.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

acknowledged writes survive the configured failure model and replicas converge. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Coordinator dies during two-phase commitResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Hot sequential key overloads one rangeIsolate the key/tenant/priority, apply admission control, and preserve durable work.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Long read blocks MVCC garbage collectionStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Leaseholder moves during transactionStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Schema version changes mid-queryStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Coordinator dies during two-phase commit
operation_id: op_01J...
aggregate_key: distributed-sql_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Observability

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Coordinator dies during two-phase commit — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

How does two-phase commit recover?

Range partitions preserve ordered scans while splitting hot ranges; hash sharding is simpler for point workloads but destroys locality. Walk the concrete lifecycle step “Parse/optimize SQL and route key spans using versioned range descriptors.” Then test it against “Coordinator dies during two-phase commit”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How are ranges split?

Consensus replicates each range; distributed transactions coordinate only the ranges touched and pay latency proportional to geography. Walk the concrete lifecycle step “Choose read timestamp and execute local or distributed reads from MVCC snapshots.” Then test it against “Hot sequential key overloads one range”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How is serializable isolation implemented?

MVCC enables consistent snapshots and online reads, but garbage collection must honor active transactions, backups, and change feeds. Walk the concrete lifecycle step “For writes acquire intents/locks, replicate each range command by consensus, and coordinate commit.” Then test it against “Long read blocks MVCC garbage collection”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

What happens to long transactions?

Range partitions preserve ordered scans while splitting hot ranges; hash sharding is simpler for point workloads but destroys locality. Walk the concrete lifecycle step “Resolve abandoned intents and publish changefeed after commit timestamp is durable.” Then test it against “Leaseholder moves during transaction”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do schema changes run online?

Consensus replicates each range; distributed transactions coordinate only the ranges touched and pay latency proportional to geography. Walk the concrete lifecycle step “Split/merge hot ranges, rebalance replicas, compact MVCC, and run online schema migrations.” Then test it against “Schema version changes mid-query”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invariantacknowledged writes survive the configured failure model and replicas converge.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects acknowledged writes survive the configured failure model and replicas converge. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top