Return relevant prefix suggestions in milliseconds while trends change continuously.
Choose a path, then play or step through exactly how the call moves between components.
The caller starts the operation at Edge Cache with a stable request and retry identity.
The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.
State assumptions before choosing technology. The supplied workload is 500K QPS · p99 < 50 ms · billions of queries. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.
| Dimension | Working estimate | Design implication |
|---|---|---|
| Traffic | 500K QPS · p99 < 50 ms · billions of queries | Separate online compute, durable state, and asynchronous consumers. |
| Peak factor | 5–10× average | Admission control, autoscaling headroom, and hot-key protection. |
| Stored state | write rate × retention × record bytes | Partition count, indexes, compaction, backup, and archival tier. |
| Network | peak operations × payload size | Batching, compression, edge placement, and egress budget. |
| Availability | 99.9–99.99% | Multi-zone replicas, tested failover, and dependency error budgets. |
| Recovery | RPO/RTO by invariant | Ack boundary, replay retention, reconciliation, and operator runbooks. |
GET /v1/suggest?q=&locale=top-k suggestionsPOST /internal/eventsquery/click aggregatePOST /v1/removalsurgent unsafe-term removalPUT /v1/snapshots/{version}atomic publishsuggestion(normalized_query, locale, frequency, recency, safety_state) snapshot(version, locale, shard_range, uri, checksum) overlay(prefix, suggestion, score_delta, expires_at) removal(term_hash, reason, effective_version)
# Command envelope
operation: GET /v1/suggest?q=&locale=
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: search-autocomplete_123
expected_version: 17
payload: {...}
# Durable event
event_id: evt_01J...
aggregate_key: search-autocomplete_123
event_type: search_autocomplete_committed
schema_version: 3
trace_id: tr_8ac...The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.
↑ topThe boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.
| Component | Owned responsibility | Why separate it | Rejected alternative | Cost and evolution trigger |
|---|---|---|---|---|
| Search Box | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Normalize Unicode, case, whitespace, and locale before routing the prefix. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Edge Cache. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Popular one-character prefix becomes hot” to the external caller. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Search Box boundary before extracting or scaling it. |
| Edge Cache | Owns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Read bounded candidates from an in-memory FST plus a small recent-trend overlay. | Removes repeated work from the latency-critical path. It prevents Search Box from also owning the contract and failure behavior of Suggestion API. | Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Unsafe term must disappear immediately” to Search Box. | Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (500K QPS · p99 < 50 ms · billions of queries) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Edge Cache boundary before extracting or scaling it. |
| Suggestion API | Owns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Merge shard results, remove unsafe/deleted terms, and optionally blend personal context. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Edge Cache from also owning the contract and failure behavior of Normalizer / Locale. | Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Snapshot versions differ across replicas” to Edge Cache. | Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Suggestion API boundary before extracting or scaling it. |
| Normalizer / Locale | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Rank by frequency, recency, edit distance, locale, and business/safety constraints. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Suggestion API from also owning the contract and failure behavior of Prefix Router. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Trend poisoning manipulates ranking” to Suggestion API. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Normalizer / Locale boundary before extracting or scaling it. |
| Prefix Router | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Publish immutable bulk snapshots atomically and expire the streaming overlay. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Normalizer / Locale from also owning the contract and failure behavior of FST Shards. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Personalization leaks prior queries” to Normalizer / Locale. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Prefix Router boundary before extracting or scaling it. |
| FST Shards | Serves a low-latency, disposable projection with authoritative fallback. In this design it participates in: Normalize Unicode, case, whitespace, and locale before routing the prefix. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Prefix Router from also owning the contract and failure behavior of Recent Overlay. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Popular one-character prefix becomes hot” to Prefix Router. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the FST Shards boundary before extracting or scaling it. |
| Recent Overlay | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Read bounded candidates from an in-memory FST plus a small recent-trend overlay. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents FST Shards from also owning the contract and failure behavior of Candidate Merger. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Unsafe term must disappear immediately” to FST Shards. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Recent Overlay boundary before extracting or scaling it. |
| Candidate Merger | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Merge shard results, remove unsafe/deleted terms, and optionally blend personal context. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Recent Overlay from also owning the contract and failure behavior of Safety Filter. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Snapshot versions differ across replicas” to Recent Overlay. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Candidate Merger boundary before extracting or scaling it. |
| Safety Filter | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Rank by frequency, recency, edit distance, locale, and business/safety constraints. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Candidate Merger from also owning the contract and failure behavior of Personalization. | Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Trend poisoning manipulates ranking” to Candidate Merger. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Safety Filter boundary before extracting or scaling it. |
| Personalization | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Publish immutable bulk snapshots atomically and expire the streaming overlay. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Safety Filter from also owning the contract and failure behavior of Top-K Ranker. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Personalization leaks prior queries” to Safety Filter. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Personalization boundary before extracting or scaling it. |
| Top-K Ranker | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Normalize Unicode, case, whitespace, and locale before routing the prefix. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Personalization from also owning the contract and failure behavior of Snapshot Builder. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Popular one-character prefix becomes hot” to Personalization. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Top-K Ranker boundary before extracting or scaling it. |
| Snapshot Builder | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Read bounded candidates from an in-memory FST plus a small recent-trend overlay. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Top-K Ranker from also owning the contract and failure behavior of Query Event Stream. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Unsafe term must disappear immediately” to Top-K Ranker. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Snapshot Builder boundary before extracting or scaling it. |
| Query Event Stream | Preserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Merge shard results, remove unsafe/deleted terms, and optionally blend personal context. | Decouples bursts and failure from user acknowledgment. It prevents Snapshot Builder from also owning the contract and failure behavior of Removal Control Plane. | Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Snapshot versions differ across replicas” to Snapshot Builder. | Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Query Event Stream boundary before extracting or scaling it. |
| Removal Control Plane | Owns one bounded domain decision, transition, or operational capability. In this design it participates in: Rank by frequency, recency, edit distance, locale, and business/safety constraints. | Separates a distinct scale, ownership, policy, or failure boundary. It prevents Query Event Stream from also owning the contract and failure behavior of operators and repair tooling. | Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Trend poisoning manipulates ranking” to Query Event Stream. | Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Removal Control Plane boundary before extracting or scaling it. |
Serve compact tries or FSTs from memory and shard by normalized prefix ranges.
A single deployable with one transactional store is the right starting point for a small search autocomplete workload. The proposed boundary becomes justified when serve compact tries or fsts from memory and shard by normalized prefix ranges. Separating it contains the named failure “Popular one-character prefix becomes hot” without placing unrelated work on the critical path.
Combine historical frequency, recency, locale, safety, and optional personalization in ranking.
The design coordinates only where each accepted search autocomplete operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.
Publish immutable snapshots for bulk updates and overlay a small streaming index for trends.
This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Merge shard results, remove unsafe/deleted terms, and optionally blend personal context.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.
Serve compact tries or FSTs from memory and shard by normalized prefix ranges.
Combine historical frequency, recency, locale, safety, and optional personalization in ranking.
Publish immutable snapshots for bulk updates and overlay a small streaming index for trends.
Normalize Unicode, case, whitespace, and locale before routing the prefix.
Read bounded candidates from an in-memory FST plus a small recent-trend overlay.
Merge shard results, remove unsafe/deleted terms, and optionally blend personal context.
Rank by frequency, recency, edit distance, locale, and business/safety constraints.
Publish immutable bulk snapshots atomically and expire the streaming overlay.
identity = authenticate(request) prior = idempotency.get(identity, request.key) if prior: return prior.result current = authoritative_store.read(partition_key) next_state, events = domain.transition(current, request) authoritative_store.compare_and_set(current.version, next_state) idempotency.commit(request.key, next_state.result) outbox.append(events) return next_state.result
Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.
Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.
Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.
Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.
partition = stable_hash(primary_or_aggregate_key) → shard owner write = conditional_update(expected_version) read_projection = versioned and rebuildable
each accepted search autocomplete operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.
Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.
| Operation | Consistency | Reason |
|---|---|---|
| Authoritative mutation | Conditional / serializable | Protect uniqueness, ownership, money, inventory, order, or legal state transition. |
| Read after own write | Primary, session token, or cache warm | The caller must observe the result it just committed. |
| Ordinary read projection | Bounded stale where safe | Replicas and caches improve latency when version/freshness is visible. |
| Analytics and operations | Eventual + replay | Freshness is less important than isolating the online path. |
| Failure | Runtime behavior | Recovery proof |
|---|---|---|
| Popular one-character prefix becomes hot | Isolate the key/tenant/priority, apply admission control, and preserve durable work. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Unsafe term must disappear immediately | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Snapshot versions differ across replicas | Isolate the key/tenant/priority, apply admission control, and preserve durable work. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Trend poisoning manipulates ranking | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
| Personalization leaks prior queries | Stop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state. | Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline. |
# Recovery checkpoint scenario: Popular one-character prefix becomes hot operation_id: op_01J... aggregate_key: search-autocomplete_123 last_durable_version: 18 projection_checkpoint: partition-07@offset-482901 retry_policy: same_identity / bounded_backoff reconciliation_status: pending owner: Removal Control Plane
Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.
↑ topServe compact tries or FSTs from memory and shard by normalized prefix ranges. Walk the concrete lifecycle step “Normalize Unicode, case, whitespace, and locale before routing the prefix.” Then test it against “Popular one-character prefix becomes hot”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Combine historical frequency, recency, locale, safety, and optional personalization in ranking. Walk the concrete lifecycle step “Read bounded candidates from an in-memory FST plus a small recent-trend overlay.” Then test it against “Unsafe term must disappear immediately”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Publish immutable snapshots for bulk updates and overlay a small streaming index for trends. Walk the concrete lifecycle step “Merge shard results, remove unsafe/deleted terms, and optionally blend personal context.” Then test it against “Snapshot versions differ across replicas”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Serve compact tries or FSTs from memory and shard by normalized prefix ranges. Walk the concrete lifecycle step “Rank by frequency, recency, edit distance, locale, and business/safety constraints.” Then test it against “Trend poisoning manipulates ranking”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
Combine historical frequency, recency, locale, safety, and optional personalization in ranking. Walk the concrete lifecycle step “Publish immutable bulk snapshots atomically and expire the streaming overlay.” Then test it against “Personalization leaks prior queries”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.
| Time | What to cover |
|---|---|
| 0–5 min | Clarify scope, workload, SLOs, and the correctness invariant. |
| 5–10 min | Estimate throughput/storage and define concrete APIs and records. |
| 10–20 min | Draw online, durable-write, and asynchronous/repair paths. |
| 20–30 min | Explain the exact lifecycle, partition key, and concurrency control. |
| 30–38 min | Discuss caching, hot keys, replication, and backpressure. |
| 38–43 min | Run failure drills, security, observability, and regional evolution. |
| 43–45 min | Summarize the five most important choices and trade-offs. |
| Invariant | each accepted search autocomplete operation has one stable result and retries do not repeat its business effect. |
|---|---|
| Partitioning | Stable aggregate/key hash with versioned ownership and measured hot-key isolation. |
| Commit | Expected version or uniqueness at the authoritative state boundary. |
| Async work | Durable outbox/log, idempotent consumers, checkpoints, and replay. |
| Recovery | Same operation identity, durable truth lookup, and business reconciliation. |