22 / 60 · Storage

Cloud Object Storage — Production Architecture

All designs

Store immutable objects with multipart upload, replication, versioning, and extreme durability.

Exabytes · 11 nines durability · objects up to TBs
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT CLOUD OBJECT STORAGE · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 SERVICE SDK/CLI domain subsystem 02 EDGE Global Front Door protocol · auth · quota 03 SECURITY Auth/Bucket Policy decision & coordination 04 SERVICE Metadata Partition domain subsystem 05 SERVICE MultipartCoordinator decision & coordination 06 SERVICE Placement Service routing & ownership 07 EDGE Chunk Gateway protocol · auth · quota 08 SERVICE Storage Nodes domain subsystem 09 SERVICE Erasure Encoder domain subsystem 10 DATA Manifest Store authoritative durable state 11 SERVICE Replication/Repair asynchronous execution 12 SERVICE Integrity Scrubber domain subsystem 13 SERVICE Lifecycle Tiering domain subsystem 14 SERVICE Event Notifications domain subsystem 15 SERVICE Range Read Cache low-latency read path externaledgeservicedatastreamsecurityops
Step 1 / 6SDK / CLI

The caller starts the operation at SDK / CLI with a stable request and retry identity.

  1. 01 · SDK / CLIThe caller starts the operation at SDK / CLI with a stable request and retry identity.
  2. 02 · Global Front DoorAuthorize bucket/key and create a version-scoped multipart upload.
  3. 03 · Auth / Bucket PolicyChecksum parts, erasure-code chunks, and place fragments across failure domains.
  4. 04 · Metadata PartitionAt complete, validate ordered parts and atomically publish an immutable manifest/version.
  5. 05 · Storage NodesRange reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.
  6. 06 · Range Read CacheScrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.
2. Critical request and recovery sequence
MUTATION / COMMITSDK/CLIACTOR 01Global FrontDoorACTOR 02MetadataPartitionACTOR 03Storage NodesACTOR 04ONLINE / READ + RECOVERYSDK/CLIACTOR 01Auth/BucketPolicyACTOR 02PlacementServiceACTOR 03Range ReadCacheACTOR 04Authorize bucket/key and create a / version-scoped multipart upload.Checksum parts, erasure-code / chunks, and place fragments acrossAt complete, validate ordered parts / and atomically publish an immutablecommitted result + eventAt complete, validate ordered parts / and atomically publish an immutableRange reads map bytes to chunks, / fetch sufficient fragments, verify,Scrub continuously, repair / under-replicated fragments, andcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

SDK / CLI

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Global Front Door

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Auth / Bucket Policy

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Metadata Partition

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary cloud object storage workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for Exabytes · 11 nines durability · objects up to TBs.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: a published version is complete, checksummed, authorized, and never silently corrupted.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is Exabytes · 11 nines durability · objects up to TBs. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
TrafficExabytes · 11 nines durability · objects up to TBsSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

POST /bucket/key?uploadsmultipart session
PUT /bucket/key?partNumber=&uploadId=part
POST /bucket/key?uploadId=atomic manifest commit
GET /bucket/key Range: bytes=versioned range read

Authoritative records and indexes

object(bucket, key, version, manifest_id, etag, state)

manifest(manifest_id, ordered_chunks, erasure_scheme, checksum)

chunk(chunk_id, placements, size, checksum)

upload(upload_id, bucket, key, parts, expires_at)

Command and event envelopes

# Command envelope
operation: POST /bucket/key?uploads
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: object-storage_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: object-storage_123
event_type: object_storage_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
SDK / CLIOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authorize bucket/key and create a version-scoped multipart upload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Global Front Door.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Manifest publishes before fragment durability” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the SDK / CLI boundary before extracting or scaling it.
Global Front DoorOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Checksum parts, erasure-code chunks, and place fragments across failure domains.Separates a distinct scale, ownership, policy, or failure boundary. It prevents SDK / CLI from also owning the contract and failure behavior of Auth / Bucket Policy.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Rack loss removes too many fragments” to SDK / CLI.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Global Front Door boundary before extracting or scaling it.
Auth / Bucket PolicyOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At complete, validate ordered parts and atomically publish an immutable manifest/version.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Global Front Door from also owning the contract and failure behavior of Metadata Partition.Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Multipart complete races abort” to Global Front Door.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Auth / Bucket Policy boundary before extracting or scaling it.
Metadata PartitionOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Range reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auth / Bucket Policy from also owning the contract and failure behavior of Multipart Coordinator.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Corrupt fragment passes disk read” to Auth / Bucket Policy.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Metadata Partition boundary before extracting or scaling it.
Multipart CoordinatorOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Scrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Metadata Partition from also owning the contract and failure behavior of Placement Service.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Delete marker conflicts with versioned read” to Metadata Partition.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Multipart Coordinator boundary before extracting or scaling it.
Placement ServiceOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authorize bucket/key and create a version-scoped multipart upload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Multipart Coordinator from also owning the contract and failure behavior of Chunk Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Manifest publishes before fragment durability” to Multipart Coordinator.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Placement Service boundary before extracting or scaling it.
Chunk GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Checksum parts, erasure-code chunks, and place fragments across failure domains.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Placement Service from also owning the contract and failure behavior of Storage Nodes.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Rack loss removes too many fragments” to Placement Service.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Chunk Gateway boundary before extracting or scaling it.
Storage NodesOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At complete, validate ordered parts and atomically publish an immutable manifest/version.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Chunk Gateway from also owning the contract and failure behavior of Erasure Encoder.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Multipart complete races abort” to Chunk Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Storage Nodes boundary before extracting or scaling it.
Erasure EncoderOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Range reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Storage Nodes from also owning the contract and failure behavior of Manifest Store.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Corrupt fragment passes disk read” to Storage Nodes.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Erasure Encoder boundary before extracting or scaling it.
Manifest StoreOwns authoritative versioned state or immutable evidence. In this design it participates in: Scrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.Defines the source of truth and commit boundary. It prevents Erasure Encoder from also owning the contract and failure behavior of Replication / Repair.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Delete marker conflicts with versioned read” to Erasure Encoder.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Manifest Store boundary before extracting or scaling it.
Replication / RepairExecutes idempotent asynchronous work under lease and retry policy. In this design it participates in: Authorize bucket/key and create a version-scoped multipart upload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Manifest Store from also owning the contract and failure behavior of Integrity Scrubber.Perform the work before acknowledging the caller. This gives simpler consistency but unbounded tail latency and fragile retries. Here that simpler choice would couple “Manifest publishes before fragment durability” to Manifest Store.Costs leases, checkpoints, retries, DLQs, and lag operations. Separate once execution time or burstiness no longer fits the request deadline. Measure the latency, saturation, lag, or mismatch at the Replication / Repair boundary before extracting or scaling it.
Integrity ScrubberOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Checksum parts, erasure-code chunks, and place fragments across failure domains.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Replication / Repair from also owning the contract and failure behavior of Lifecycle Tiering.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Rack loss removes too many fragments” to Replication / Repair.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Integrity Scrubber boundary before extracting or scaling it.
Lifecycle TieringOwns one bounded domain decision, transition, or operational capability. In this design it participates in: At complete, validate ordered parts and atomically publish an immutable manifest/version.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Integrity Scrubber from also owning the contract and failure behavior of Event Notifications.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Multipart complete races abort” to Integrity Scrubber.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Lifecycle Tiering boundary before extracting or scaling it.
Event NotificationsOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Range reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Lifecycle Tiering from also owning the contract and failure behavior of Range Read Cache.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Corrupt fragment passes disk read” to Lifecycle Tiering.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Event Notifications boundary before extracting or scaling it.
Range Read CacheServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Scrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.Removes repeated work from the latency-critical path. It prevents Event Notifications from also owning the contract and failure behavior of operators and repair tooling.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Delete marker conflicts with versioned read” to Event Notifications.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (Exabytes · 11 nines durability · objects up to TBs) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Range Read Cache boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Separate metadata from bulk data; place erasure-coded chunks across racks and zones.

A single deployable with one transactional store is the right starting point for a small cloud object storage workload. The proposed boundary becomes justified when separate metadata from bulk data; place erasure-coded chunks across racks and zones. Separating it contains the named failure “Manifest publishes before fragment durability” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Use multipart uploads with checksums, immutable object versions, and atomic manifest commit.

The design coordinates only where a published version is complete, checksummed, authorized, and never silently corrupted. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Continuously scrub data, repair missing fragments, and verify durability through independent metadata.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “At complete, validate ordered parts and atomically publish an immutable manifest/version.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect a published version is complete, checksummed, authorized, and never silently corrupted. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Separate metadata from bulk data; place erasure-coded chunks across racks and zones.

Scale boundary

Use multipart uploads with checksums, immutable object versions, and atomic manifest commit.

Operational boundary

Continuously scrub data, repair missing fragments, and verify durability through independent metadata.

↑ top

5. Critical flow and algorithm

01

Authorize bucket/key and create a version-scoped multipart upload.

02

Checksum parts, erasure-code chunks, and place fragments across failure domains.

03

At complete, validate ordered parts and atomically publish an immutable manifest/version.

04

Range reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.

05

Scrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

a published version is complete, checksummed, authorized, and never silently corrupted. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Manifest publishes before fragment durabilityStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Rack loss removes too many fragmentsStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Multipart complete races abortResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Corrupt fragment passes disk readStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Delete marker conflicts with versioned readResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Manifest publishes before fragment durability
operation_id: op_01J...
aggregate_key: object-storage_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Range Read Cache

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Manifest publishes before fragment durability — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

Replication or erasure coding?

Separate metadata from bulk data; place erasure-coded chunks across racks and zones. Walk the concrete lifecycle step “Authorize bucket/key and create a version-scoped multipart upload.” Then test it against “Manifest publishes before fragment durability”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How does multipart commit work?

Use multipart uploads with checksums, immutable object versions, and atomic manifest commit. Walk the concrete lifecycle step “Checksum parts, erasure-code chunks, and place fragments across failure domains.” Then test it against “Rack loss removes too many fragments”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do range reads work?

Continuously scrub data, repair missing fragments, and verify durability through independent metadata. Walk the concrete lifecycle step “At complete, validate ordered parts and atomically publish an immutable manifest/version.” Then test it against “Multipart complete races abort”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How are deletes and versions modeled?

Separate metadata from bulk data; place erasure-coded chunks across racks and zones. Walk the concrete lifecycle step “Range reads map bytes to chunks, fetch sufficient fragments, verify, and optionally cache.” Then test it against “Corrupt fragment passes disk read”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do lifecycle transitions work?

Use multipart uploads with checksums, immutable object versions, and atomic manifest commit. Walk the concrete lifecycle step “Scrub continuously, repair under-replicated fragments, and tier versions by lifecycle policy.” Then test it against “Delete marker conflicts with versioned read”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invarianta published version is complete, checksummed, authorized, and never silently corrupted.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects a published version is complete, checksummed, authorized, and never silently corrupted. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top