Serve low-latency reads with partitioning, eviction, replication, and predictable failure behavior.
Choose a path, then play or step through exactly how the call moves between components.
The caller starts the operation at Client Library with a stable request and retry identity.
The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.
State assumptions before choosing technology. The supplied workload is 100M ops/s · sub-ms hit · TBs of memory. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.
| Dimension | Working estimate | Design implication |
|---|---|---|
| Traffic | 100M ops/s · sub-ms hit · TBs of memory | Separate online compute, durable state, and asynchronous consumers. |
| Peak factor | 5–10× average | Admission control, autoscaling headroom, and hot-key protection. |
| Stored state | write rate × retention × record bytes | Partition count, indexes, compaction, backup, and archival tier. |
| Network | peak operations × payload size | Batching, compression, edge placement, and egress budget. |
| Availability | 99.9–99.99% | Multi-zone replicas, tested failover, and dependency error budgets. |
| Recovery | RPO/RTO by invariant | Ack boundary, replay retention, reconciliation, and operator runbooks. |
GET keyrouted lookup with version metadataSET key value ttlprimary then replica policyDELETE keytombstone/invalidationGET /admin/ringmembership generationentry(key_hash PK, value, version, expires_at, size) node(node_id, tokens, zone, generation, state) replication(key_hash, version, operation) hot_key(key_hash, qps, miss_rate, sampled_at)
# Command envelope
operation: GET key
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: distributed-cache_123
expected_version: 17
payload: {...}
# Durable event
event_id: evt_01J...
aggregate_key: distributed-cache_123
event_type: distributed_cache_committed
schema_version: 3
trace_id: tr_8ac...The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.
↑ topThe boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.
| Component | Owned responsibility | Why separate it | Rejected alternative | Cost and evolution trigger |
|---|---|---|---|---|
| Application | Owns client protocol, request identity, retry, and resume behavior. In this design it participates in: Hash the key against a versioned virtual-node ring in the client. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Client Library. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Node loss causes synchronized origin misses” to the external caller. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Application boundary before extracting or scaling it. |
| Client Library | Owns client protocol, request identity, retry, and resume behavior. In this design it participates in: Read primary or permitted replica; lazily expire and promote recency metadata. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Application from also owning the contract and failure behavior of Key Hasher. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “One key exceeds one node network capacity” to Application. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Client Library boundary before extracting or scaling it. |
| Key Hasher | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: On miss, coalesce loaders so one caller fetches origin and populates the value. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Client Library from also owning the contract and failure behavior of Membership Snapshot. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Rebalance evicts the working set” to Client Library. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Key Hasher boundary before extracting or scaling it. |
| Membership Snapshot | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Replicate according to durability mode and acknowledge at the chosen boundary. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Key Hasher from also owning the contract and failure behavior of Primary Cache Node. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Stale replica serves deleted value” to Key Hasher. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Membership Snapshot boundary before extracting or scaling it. |
| Primary Cache Node | Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: During rebalance, dual-read old/new owners and stream ranges before ownership cutover. | Removes repeated work from the latency-critical path. It prevents Membership Snapshot from also owning the contract and failure behavior of Replica Cache Node. | Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Oversized values fragment memory” to Membership Snapshot. | Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (100M ops/s · sub-ms hit · TBs of memory) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Primary Cache Node boundary before extracting or scaling it. |
| Replica Cache Node | Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: Hash the key against a versioned virtual-node ring in the client. | Removes repeated work from the latency-critical path. It prevents Primary Cache Node from also owning the contract and failure behavior of Memory Slabs. | Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Node loss causes synchronized origin misses” to Primary Cache Node. | Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (100M ops/s · sub-ms hit · TBs of memory) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Replica Cache Node boundary before extracting or scaling it. |
| Memory Slabs | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Read primary or permitted replica; lazily expire and promote recency metadata. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Replica Cache Node from also owning the contract and failure behavior of Eviction Engine. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “One key exceeds one node network capacity” to Replica Cache Node. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Memory Slabs boundary before extracting or scaling it. |
| Eviction Engine | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: On miss, coalesce loaders so one caller fetches origin and populates the value. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Memory Slabs from also owning the contract and failure behavior of TTL Wheel. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Rebalance evicts the working set” to Memory Slabs. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Eviction Engine boundary before extracting or scaling it. |
| TTL Wheel | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Replicate according to durability mode and acknowledge at the chosen boundary. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Eviction Engine from also owning the contract and failure behavior of Replication Stream. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Stale replica serves deleted value” to Eviction Engine. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the TTL Wheel boundary before extracting or scaling it. |
| Replication Stream | Preserves durable ordering, replay, and consumer checkpoints. In this design it participates in: During rebalance, dual-read old/new owners and stream ranges before ownership cutover. | Decouples bursts and failure from user acknowledgment. It prevents TTL Wheel from also owning the contract and failure behavior of Rebalancer. | Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Oversized values fragment memory” to TTL Wheel. | Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Replication Stream boundary before extracting or scaling it. |
| Rebalancer | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Hash the key against a versioned virtual-node ring in the client. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Replication Stream from also owning the contract and failure behavior of Hot-Key Detector. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Node loss causes synchronized origin misses” to Replication Stream. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Rebalancer boundary before extracting or scaling it. |
| Hot-Key Detector | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Read primary or permitted replica; lazily expire and promote recency metadata. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Rebalancer from also owning the contract and failure behavior of Origin Loader. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “One key exceeds one node network capacity” to Rebalancer. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Hot-Key Detector boundary before extracting or scaling it. |
| Origin Loader | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: On miss, coalesce loaders so one caller fetches origin and populates the value. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Hot-Key Detector from also owning the contract and failure behavior of Cache Metrics. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Rebalance evicts the working set” to Hot-Key Detector. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Origin Loader boundary before extracting or scaling it. |
| Cache Metrics | Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: Replicate according to durability mode and acknowledge at the chosen boundary. | Proves correctness and supports controlled recovery. It prevents Origin Loader from also owning the contract and failure behavior of operators and repair tooling. | Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Stale replica serves deleted value” to Origin Loader. | Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (100M ops/s · sub-ms hit · TBs of memory) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Cache Metrics boundary before extracting or scaling it. |
Use consistent hashing with virtual nodes to limit movement as capacity changes.
A single deployable with one transactional store is the right starting point for a small distributed cache workload. The proposed boundary becomes justified when use consistent hashing with virtual nodes to limit movement as capacity changes. Separating it contains the named failure “Node loss causes synchronized origin misses” without placing unrelated work on the critical path.
Choose eviction by workload—LRU/LFU/TTL—and protect against hot keys and cache stampedes.
The design coordinates only where each accepted distributed cache operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.
Replication improves availability but invalidation and stale reads remain application-level decisions.
This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “On miss, coalesce loaders so one caller fetches origin and populates the value.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.
Use consistent hashing with virtual nodes to limit movement as capacity changes.
Choose eviction by workload—LRU/LFU/TTL—and protect against hot keys and cache stampedes.
Replication improves availability but invalidation and stale reads remain application-level decisions.
Hash the key against a versioned virtual-node ring in the client.
Read primary or permitted replica; lazily expire and promote recency metadata.
On miss, coalesce loaders so one caller fetches origin and populates the value.
Replicate according to durability mode and acknowledge at the chosen boundary.
During rebalance, dual-read old/new owners and stream ranges before ownership cutover.
identity = authenticate(request) prior = idempotency.get(identity, request.key) if prior: return prior.result current = authoritative_store.read(partition_key) next_state, events = domain.transition(current, request) authoritative_store.compare_and_set(current.version, next_state) idempotency.commit(request.key, next_state.result) outbox.append(events) return next_state.result
Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.
Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.
Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.
Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.
partition = stable_hash(primary_or_aggregate_key) → shard owner write = conditional_update(expected_version) read_projection = versioned and rebuildable
each accepted distributed cache operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.
Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.
| Operation | Consistency | Reason |
|---|---|---|
| Authoritative mutation | Conditional / serializable | Protect uniqueness, ownership, money, inventory, order, or legal state transition. |
| Read after own write | Primary, session token, or cache warm | The caller must observe the result it just committed. |
| Ordinary read projection | Bounded stale where safe | Replicas and caches improve latency when version/freshness is visible. |
| Analytics and operations | Eventual + replay | Freshness is less important than isolating the online path. |
| Failure | Runtime behavior | Recovery proof |
|---|---|---|
| Node loss causes synchronized origin misses | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| One key exceeds one node network capacity | Resolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Rebalance evicts the working set | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Stale replica serves deleted value | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Oversized values fragment memory | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
# Recovery checkpoint scenario: Node loss causes synchronized origin misses operation_id: op_01J... aggregate_key: distributed-cache_123 last_durable_version: 18 projection_checkpoint: partition-07@offset-482901 retry_policy: same_identity / bounded_backoff reconciliation_status: pending owner: Cache Metrics
Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.
↑ topUse consistent hashing with virtual nodes to limit movement as capacity changes. Walk the concrete lifecycle step “Hash the key against a versioned virtual-node ring in the client.” Then test it against “Node loss causes synchronized origin misses”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Choose eviction by workload—LRU/LFU/TTL—and protect against hot keys and cache stampedes. Walk the concrete lifecycle step “Read primary or permitted replica; lazily expire and promote recency metadata.” Then test it against “One key exceeds one node network capacity”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Replication improves availability but invalidation and stale reads remain application-level decisions. Walk the concrete lifecycle step “On miss, coalesce loaders so one caller fetches origin and populates the value.” Then test it against “Rebalance evicts the working set”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Use consistent hashing with virtual nodes to limit movement as capacity changes. Walk the concrete lifecycle step “Replicate according to durability mode and acknowledge at the chosen boundary.” Then test it against “Stale replica serves deleted value”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Choose eviction by workload—LRU/LFU/TTL—and protect against hot keys and cache stampedes. Walk the concrete lifecycle step “During rebalance, dual-read old/new owners and stream ranges before ownership cutover.” Then test it against “Oversized values fragment memory”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
| Time | What to cover |
|---|---|
| 0–5 min | Clarify scope, workload, SLOs, and the correctness invariant. |
| 5–10 min | Estimate throughput/storage and define concrete APIs and records. |
| 10–20 min | Draw online, durable-write, and asynchronous/repair paths. |
| 20–30 min | Explain the exact lifecycle, partition key, and concurrency control. |
| 30–38 min | Discuss caching, hot keys, replication, and backpressure. |
| 38–43 min | Run failure drills, security, observability, and regional evolution. |
| 43–45 min | Summarize the five most important choices and trade-offs. |
| Invariant | each accepted distributed cache operation has one stable result and retries do not repeat its business effect. |
|---|---|
| Partitioning | Stable aggregate/key hash with versioned ownership and measured hot-key isolation. |
| Commit | Expected version or uniqueness at the authoritative state boundary. |
| Async work | Durable outbox/log, idempotent consumers, checkpoints, and replay. |
| Recovery | Same operation identity, durable truth lookup, and business reconciliation. |