Generate ranked, paginated feeds while absorbing celebrity fan-out.
Choose a path, then play or step through exactly how the call moves between components.
The caller starts the operation at Mobile Client with a stable request and retry identity.
The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.
State assumptions before choosing technology. The supplied workload is 500M users · 100K feed reads/s · 20K posts/s. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.
| Dimension | Working estimate | Design implication |
|---|---|---|
| Traffic | 500M users · 100K feed reads/s · 20K posts/s | Separate online compute, durable state, and asynchronous consumers. |
| Peak factor | 5–10× average | Admission control, autoscaling headroom, and hot-key protection. |
| Stored state | write rate × retention × record bytes | Partition count, indexes, compaction, backup, and archival tier. |
| Network | peak operations × payload size | Batching, compression, edge placement, and egress budget. |
| Availability | 99.9–99.99% | Multi-zone replicas, tested failover, and dependency error budgets. |
| Recovery | RPO/RTO by invariant | Ack boundary, replay retention, reconciliation, and operator runbooks. |
POST /v1/postscommit source postGET /v1/feed?cursor=ranked pageDELETE /v1/posts/{id}tombstone and invalidatePOST /v1/impressionsbatched exposure eventspost(post_id PK, author_id, body_ref, visibility, created_at) timeline(user_id, rank_key, post_id, source, inserted_at) follow(follower_id, followee_id, state) cursor(snapshot_time, rank_key, post_id)
# Command envelope
operation: POST /v1/posts
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: news-feed_123
expected_version: 17
payload: {...}
# Durable event
event_id: evt_01J...
aggregate_key: news-feed_123
event_type: news_feed_committed
schema_version: 3
trace_id: tr_8ac...The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.
↑ topThe boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.
| Component | Owned responsibility | Why separate it | Rejected alternative | Cost and evolution trigger |
|---|---|---|---|---|
| Mobile Client | Owns client protocol, request identity, retry, and resume behavior. In this design it participates in: Commit the post and emit one immutable post-created event. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Feed API. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Celebrity post creates billion-edge fan-out” to the external caller. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Mobile Client boundary before extracting or scaling it. |
| Feed API | Owns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Fan out ordinary authors into follower timeline partitions; mark celebrities pull-on-read. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Mobile Client from also owning the contract and failure behavior of Auth / Policy. | Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Feed cursor sees inserts between pages” to Mobile Client. | Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Feed API boundary before extracting or scaling it. |
| Auth / Policy | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: At read, merge materialized timeline, celebrity candidates, and eligible ads. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Feed API from also owning the contract and failure behavior of Candidate Service. | Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Deleted private post remains cached” to Feed API. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Auth / Policy boundary before extracting or scaling it. |
| Candidate Service | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Filter privacy/deletes, fetch features, rank a bounded candidate set, and paginate by cursor. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Auth / Policy from also owning the contract and failure behavior of Social Graph. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Ranking service exceeds latency budget” to Auth / Policy. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Candidate Service boundary before extracting or scaling it. |
| Social Graph | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Log impressions asynchronously with model and candidate-set versions. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Candidate Service from also owning the contract and failure behavior of Post Store. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Fanout lag causes stale timelines” to Candidate Service. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Social Graph boundary before extracting or scaling it. |
| Post Store | Owns authoritative versioned state or immutable evidence. In this design it participates in: Commit the post and emit one immutable post-created event. | Defines the source of truth and commit boundary. It prevents Social Graph from also owning the contract and failure behavior of Fanout Router. | Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Celebrity post creates billion-edge fan-out” to Social Graph. | Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Post Store boundary before extracting or scaling it. |
| Fanout Router | Executes idempotent asynchronous work under lease and retry policy. In this design it participates in: Fan out ordinary authors into follower timeline partitions; mark celebrities pull-on-read. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Post Store from also owning the contract and failure behavior of Timeline Workers. | Perform the work before acknowledging the caller. This gives simpler consistency but unbounded tail latency and fragile retries. Here that simpler choice would couple “Feed cursor sees inserts between pages” to Post Store. | Costs leases, checkpoints, retries, DLQs, and lag operations. Separate once execution time or burstiness no longer fits the request deadline. Measure the latency, saturation, lag, or mismatch at the Fanout Router boundary before extracting or scaling it. |
| Timeline Workers | Executes idempotent asynchronous work under lease and retry policy. In this design it participates in: At read, merge materialized timeline, celebrity candidates, and eligible ads. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Fanout Router from also owning the contract and failure behavior of Timeline Cache. | Perform the work before acknowledging the caller. This gives simpler consistency but unbounded tail latency and fragile retries. Here that simpler choice would couple “Deleted private post remains cached” to Fanout Router. | Costs leases, checkpoints, retries, DLQs, and lag operations. Separate once execution time or burstiness no longer fits the request deadline. Measure the latency, saturation, lag, or mismatch at the Timeline Workers boundary before extracting or scaling it. |
| Timeline Cache | Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: Filter privacy/deletes, fetch features, rank a bounded candidate set, and paginate by cursor. | Removes repeated work from the latency-critical path. It prevents Timeline Workers from also owning the contract and failure behavior of Celebrity Pull Path. | Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Ranking service exceeds latency budget” to Timeline Workers. | Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (500M users · 100K feed reads/s · 20K posts/s) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Timeline Cache boundary before extracting or scaling it. |
| Celebrity Pull Path | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Log impressions asynchronously with model and candidate-set versions. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Timeline Cache from also owning the contract and failure behavior of Ranking Service. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Fanout lag causes stale timelines” to Timeline Cache. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Celebrity Pull Path boundary before extracting or scaling it. |
| Ranking Service | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Commit the post and emit one immutable post-created event. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Celebrity Pull Path from also owning the contract and failure behavior of Feature Store. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Celebrity post creates billion-edge fan-out” to Celebrity Pull Path. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Ranking Service boundary before extracting or scaling it. |
| Feature Store | Owns authoritative versioned state or immutable evidence. In this design it participates in: Fan out ordinary authors into follower timeline partitions; mark celebrities pull-on-read. | Defines the source of truth and commit boundary. It prevents Ranking Service from also owning the contract and failure behavior of Ads Mixer. | Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Feed cursor sees inserts between pages” to Ranking Service. | Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Feature Store boundary before extracting or scaling it. |
| Ads Mixer | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: At read, merge materialized timeline, celebrity candidates, and eligible ads. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Feature Store from also owning the contract and failure behavior of Impression Stream. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Deleted private post remains cached” to Feature Store. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Ads Mixer boundary before extracting or scaling it. |
| Impression Stream | Preserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Filter privacy/deletes, fetch features, rank a bounded candidate set, and paginate by cursor. | Decouples bursts and failure from user acknowledgment. It prevents Ads Mixer from also owning the contract and failure behavior of operators and repair tooling. | Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Ranking service exceeds latency budget” to Ads Mixer. | Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Impression Stream boundary before extracting or scaling it. |
Use fan-out-on-write for ordinary authors and fan-out-on-read for celebrities.
A single deployable with one transactional store is the right starting point for a small news feed workload. The proposed boundary becomes justified when use fan-out-on-write for ordinary authors and fan-out-on-read for celebrities. Separating it contains the named failure “Celebrity post creates billion-edge fan-out” without placing unrelated work on the critical path.
Store stable cursor tokens rather than offsets; rank candidates after filtering and policy checks.
The design coordinates only where each accepted news feed operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.
Separate candidate generation from ranking so models can evolve without changing feed storage.
This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “At read, merge materialized timeline, celebrity candidates, and eligible ads.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.
Use fan-out-on-write for ordinary authors and fan-out-on-read for celebrities.
Store stable cursor tokens rather than offsets; rank candidates after filtering and policy checks.
Separate candidate generation from ranking so models can evolve without changing feed storage.
Commit the post and emit one immutable post-created event.
Fan out ordinary authors into follower timeline partitions; mark celebrities pull-on-read.
At read, merge materialized timeline, celebrity candidates, and eligible ads.
Filter privacy/deletes, fetch features, rank a bounded candidate set, and paginate by cursor.
Log impressions asynchronously with model and candidate-set versions.
identity = authenticate(request) prior = idempotency.get(identity, request.key) if prior: return prior.result current = authoritative_store.read(partition_key) next_state, events = domain.transition(current, request) authoritative_store.compare_and_set(current.version, next_state) idempotency.commit(request.key, next_state.result) outbox.append(events) return next_state.result
Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.
Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.
Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.
Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.
partition = stable_hash(primary_or_aggregate_key) → shard owner write = conditional_update(expected_version) read_projection = versioned and rebuildable
each accepted news feed operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.
Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.
| Operation | Consistency | Reason |
|---|---|---|
| Authoritative mutation | Conditional / serializable | Protect uniqueness, ownership, money, inventory, order, or legal state transition. |
| Read after own write | Primary, session token, or cache warm | The caller must observe the result it just committed. |
| Ordinary read projection | Bounded stale where safe | Replicas and caches improve latency when version/freshness is visible. |
| Analytics and operations | Eventual + replay | Freshness is less important than isolating the online path. |
| Failure | Runtime behavior | Recovery proof |
|---|---|---|
| Celebrity post creates billion-edge fan-out | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Feed cursor sees inserts between pages | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Deleted private post remains cached | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Ranking service exceeds latency budget | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Fanout lag causes stale timelines | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
# Recovery checkpoint scenario: Celebrity post creates billion-edge fan-out operation_id: op_01J... aggregate_key: news-feed_123 last_durable_version: 18 projection_checkpoint: partition-07@offset-482901 retry_policy: same_identity / bounded_backoff reconciliation_status: pending owner: Impression Stream
Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.
↑ topUse fan-out-on-write for ordinary authors and fan-out-on-read for celebrities. Walk the concrete lifecycle step “Commit the post and emit one immutable post-created event.” Then test it against “Celebrity post creates billion-edge fan-out”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Store stable cursor tokens rather than offsets; rank candidates after filtering and policy checks. Walk the concrete lifecycle step “Fan out ordinary authors into follower timeline partitions; mark celebrities pull-on-read.” Then test it against “Feed cursor sees inserts between pages”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Separate candidate generation from ranking so models can evolve without changing feed storage. Walk the concrete lifecycle step “At read, merge materialized timeline, celebrity candidates, and eligible ads.” Then test it against “Deleted private post remains cached”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Use fan-out-on-write for ordinary authors and fan-out-on-read for celebrities. Walk the concrete lifecycle step “Filter privacy/deletes, fetch features, rank a bounded candidate set, and paginate by cursor.” Then test it against “Ranking service exceeds latency budget”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Store stable cursor tokens rather than offsets; rank candidates after filtering and policy checks. Walk the concrete lifecycle step “Log impressions asynchronously with model and candidate-set versions.” Then test it against “Fanout lag causes stale timelines”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
| Time | What to cover |
|---|---|
| 0–5 min | Clarify scope, workload, SLOs, and the correctness invariant. |
| 5–10 min | Estimate throughput/storage and define concrete APIs and records. |
| 10–20 min | Draw online, durable-write, and asynchronous/repair paths. |
| 20–30 min | Explain the exact lifecycle, partition key, and concurrency control. |
| 30–38 min | Discuss caching, hot keys, replication, and backpressure. |
| 38–43 min | Run failure drills, security, observability, and regional evolution. |
| 43–45 min | Summarize the five most important choices and trade-offs. |
| Invariant | each accepted news feed operation has one stable result and retries do not repeat its business effect. |
|---|---|
| Partitioning | Stable aggregate/key hash with versioned ownership and measured hot-key isolation. |
| Commit | Expected version or uniqueness at the authoritative state boundary. |
| Async work | Durable outbox/log, idempotent consumers, checkpoints, and replay. |
| Recovery | Same operation identity, durable truth lookup, and business reconciliation. |