44 / 60 · Advertising

Advertisement Serving — Production Architecture

All designs

Select eligible ads, pace budgets, and record billable events within a strict latency budget.

1M requests/s · p99 < 80 ms · billions of campaigns
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT ADVERTISEMENT SERVING · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 SERVICE Publisher SDK domain subsystem 02 EDGE Ad Gateway protocol · auth · quota 03 SECURITY Consent/Policy decision & coordination 04 SERVICE Context Builder domain subsystem 05 SERVICE Eligibility Index low-latency read path 06 DATA Frequency Cap Store authoritative durable state 07 SERVICE Budget Lease Cache low-latency read path 08 SERVICE Pacing Service domain subsystem 09 SERVICE Candidate Retriever low-latency read path 10 SERVICE Auction Ranker domain subsystem 11 EDGE Creative CDN domain subsystem 12 SERVICE Impression Signer domain subsystem 13 STREAM Event Stream durable ordered events 14 DATA Billing Ledger protocol · auth · quota 15 SECURITY Fraud/Attribution domain subsystem externaledgeservicedatastreamsecurityops
Step 1 / 6Ad Gateway

The caller starts the operation at Ad Gateway with a stable request and retry identity.

  1. 01 · Ad GatewayThe caller starts the operation at Ad Gateway with a stable request and retry identity.
  2. 02 · Context BuilderValidate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.
  3. 03 · Consent / PolicyApply frequency caps and acquire approximate budget leases before ranking.
  4. 04 · Frequency Cap StoreRun auction/ranking, choose creative, and return signed impression metadata.
  5. 05 · Billing LedgerIngest deduplicated impression/click events and update exact billing ledger.
  6. 06 · Event StreamReconcile leases against ledger and feed fraud, attribution, and campaign reporting.
2. Critical request and recovery sequence
MUTATION / COMMITPublisher SDKACTOR 01Ad GatewayACTOR 02ContextBuilderACTOR 03Pacing ServiceACTOR 04ONLINE / READ + RECOVERYPublisher SDKACTOR 01Consent/PolicyACTOR 02Frequency CapStoreACTOR 03Fraud/AttributionACTOR 04Validate consent, slot, context, / and retrieve eligible campaign IDsApply frequency caps and acquire / approximate budget leases beforeRun auction/ranking, choose / creative, and return signedcommitted result + eventRun auction/ranking, choose / creative, and return signedIngest deduplicated / impression/click events and updateReconcile leases against ledger and / feed fraud, attribution, andcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

Publisher SDK

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Ad Gateway

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Consent / Policy

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Context Builder

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary advertisement serving workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for 1M requests/s · p99 < 80 ms · billions of campaigns.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: each accepted advertisement serving operation has one stable result and retries do not repeat its business effect.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 1M requests/s · p99 < 80 ms · billions of campaigns. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
Traffic1M requests/s · p99 < 80 ms · billions of campaignsSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

GET /v1/ad?slot=&context=select creative
POST /v1/impressionssigned exposure
POST /v1/clickssigned click
PUT /v1/campaigns/{id}versioned campaign

Authoritative records and indexes

campaign(campaign_id PK, targeting, bid, budget, state)

budget(campaign_id, day, spent, reserved, version)

frequency(user_hash, campaign_id, count, expires_at)

ad_event(event_id, request_id, type, signature, occurred_at)

Command and event envelopes

# Command envelope
operation: GET /v1/ad?slot=&context=
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: ad-serving_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: ad-serving_123
event_type: ad_serving_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
Publisher SDKOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Validate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Ad Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Hot campaign budget lease overspends” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Publisher SDK boundary before extracting or scaling it.
Ad GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Apply frequency caps and acquire approximate budget leases before ranking.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Publisher SDK from also owning the contract and failure behavior of Consent / Policy.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Duplicate impression reaches billing” to Publisher SDK.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Ad Gateway boundary before extracting or scaling it.
Consent / PolicyOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Run auction/ranking, choose creative, and return signed impression metadata.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Ad Gateway from also owning the contract and failure behavior of Context Builder.Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Creative is removed while cached” to Ad Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Consent / Policy boundary before extracting or scaling it.
Context BuilderOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Ingest deduplicated impression/click events and update exact billing ledger.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Consent / Policy from also owning the contract and failure behavior of Eligibility Index.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Frequency cap store unavailable” to Consent / Policy.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Context Builder boundary before extracting or scaling it.
Eligibility IndexServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Reconcile leases against ledger and feed fraud, attribution, and campaign reporting.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Context Builder from also owning the contract and failure behavior of Frequency Cap Store.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Fraud campaign generates synthetic clicks” to Context Builder.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (1M requests/s · p99 < 80 ms · billions of campaigns) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Eligibility Index boundary before extracting or scaling it.
Frequency Cap StoreOwns authoritative versioned state or immutable evidence. In this design it participates in: Validate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.Defines the source of truth and commit boundary. It prevents Eligibility Index from also owning the contract and failure behavior of Budget Lease Cache.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Hot campaign budget lease overspends” to Eligibility Index.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Frequency Cap Store boundary before extracting or scaling it.
Budget Lease CacheServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Apply frequency caps and acquire approximate budget leases before ranking.Removes repeated work from the latency-critical path. It prevents Frequency Cap Store from also owning the contract and failure behavior of Pacing Service.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Duplicate impression reaches billing” to Frequency Cap Store.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (1M requests/s · p99 < 80 ms · billions of campaigns) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Budget Lease Cache boundary before extracting or scaling it.
Pacing ServiceOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Run auction/ranking, choose creative, and return signed impression metadata.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Budget Lease Cache from also owning the contract and failure behavior of Candidate Retriever.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Creative is removed while cached” to Budget Lease Cache.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Pacing Service boundary before extracting or scaling it.
Candidate RetrieverServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Ingest deduplicated impression/click events and update exact billing ledger.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Pacing Service from also owning the contract and failure behavior of Auction Ranker.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Frequency cap store unavailable” to Pacing Service.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Candidate Retriever boundary before extracting or scaling it.
Auction RankerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Reconcile leases against ledger and feed fraud, attribution, and campaign reporting.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Candidate Retriever from also owning the contract and failure behavior of Creative CDN.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Fraud campaign generates synthetic clicks” to Candidate Retriever.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Auction Ranker boundary before extracting or scaling it.
Creative CDNOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Validate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.Removes repeated work from the latency-critical path. It prevents Auction Ranker from also owning the contract and failure behavior of Impression Signer.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Hot campaign budget lease overspends” to Auction Ranker.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (1M requests/s · p99 < 80 ms · billions of campaigns) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Creative CDN boundary before extracting or scaling it.
Impression SignerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Apply frequency caps and acquire approximate budget leases before ranking.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Creative CDN from also owning the contract and failure behavior of Event Stream.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Duplicate impression reaches billing” to Creative CDN.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Impression Signer boundary before extracting or scaling it.
Event StreamPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Run auction/ranking, choose creative, and return signed impression metadata.Decouples bursts and failure from user acknowledgment. It prevents Impression Signer from also owning the contract and failure behavior of Billing Ledger.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Creative is removed while cached” to Impression Signer.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Event Stream boundary before extracting or scaling it.
Billing LedgerOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Ingest deduplicated impression/click events and update exact billing ledger.Defines the source of truth and commit boundary. It prevents Event Stream from also owning the contract and failure behavior of Fraud / Attribution.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Frequency cap store unavailable” to Event Stream.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Billing Ledger boundary before extracting or scaling it.
Fraud / AttributionOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Reconcile leases against ledger and feed fraud, attribution, and campaign reporting.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Billing Ledger from also owning the contract and failure behavior of operators and repair tooling.Embed policy in every caller. Rules then drift by deployment version and emergency changes require fleet-wide redeployment. Here that simpler choice would couple “Fraud campaign generates synthetic clicks” to Billing Ledger.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Fraud / Attribution boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Precompute campaign eligibility into memory indexes; querying the campaign database on every impression cannot meet latency or QPS.

A single deployable with one transactional store is the right starting point for a small advertisement serving workload. The proposed boundary becomes justified when precompute campaign eligibility into memory indexes; querying the campaign database on every impression cannot meet latency or qps. Separating it contains the named failure “Hot campaign budget lease overspends” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Reserve budget approximately on the hot path and reconcile exactly in the ledger; global serial counters would destroy availability.

The design coordinates only where each accepted advertisement serving operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Separate selection from billing evidence: clicks and impressions are signed, deduplicated events processed under fraud and attribution windows.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Run auction/ranking, choose creative, and return signed impression metadata.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect each accepted advertisement serving operation has one stable result and retries do not repeat its business effect. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Precompute campaign eligibility into memory indexes; querying the campaign database on every impression cannot meet latency or QPS.

Scale boundary

Reserve budget approximately on the hot path and reconcile exactly in the ledger; global serial counters would destroy availability.

Operational boundary

Separate selection from billing evidence: clicks and impressions are signed, deduplicated events processed under fraud and attribution windows.

↑ top

5. Critical flow and algorithm

01

Validate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.

02

Apply frequency caps and acquire approximate budget leases before ranking.

03

Run auction/ranking, choose creative, and return signed impression metadata.

04

Ingest deduplicated impression/click events and update exact billing ledger.

05

Reconcile leases against ledger and feed fraud, attribution, and campaign reporting.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

each accepted advertisement serving operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Hot campaign budget lease overspendsIsolate the key/tenant/priority, apply admission control, and preserve durable work.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Duplicate impression reaches billingResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Creative is removed while cachedStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Frequency cap store unavailableStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Fraud campaign generates synthetic clicksStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Hot campaign budget lease overspends
operation_id: op_01J...
aggregate_key: ad-serving_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Fraud / Attribution

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Hot campaign budget lease overspends — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

How do you pace a daily budget?

Precompute campaign eligibility into memory indexes; querying the campaign database on every impression cannot meet latency or QPS. Walk the concrete lifecycle step “Validate consent, slot, context, and retrieve eligible campaign IDs from memory indexes.” Then test it against “Hot campaign budget lease overspends”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How is an auction ranked?

Reserve budget approximately on the hot path and reconcile exactly in the ledger; global serial counters would destroy availability. Walk the concrete lifecycle step “Apply frequency caps and acquire approximate budget leases before ranking.” Then test it against “Duplicate impression reaches billing”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you avoid overspend?

Separate selection from billing evidence: clicks and impressions are signed, deduplicated events processed under fraud and attribution windows. Walk the concrete lifecycle step “Run auction/ranking, choose creative, and return signed impression metadata.” Then test it against “Creative is removed while cached”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How is frequency capping stored?

Precompute campaign eligibility into memory indexes; querying the campaign database on every impression cannot meet latency or QPS. Walk the concrete lifecycle step “Ingest deduplicated impression/click events and update exact billing ledger.” Then test it against “Frequency cap store unavailable”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How are fraudulent clicks handled?

Reserve budget approximately on the hot path and reconcile exactly in the ledger; global serial counters would destroy availability. Walk the concrete lifecycle step “Reconcile leases against ledger and feed fraud, attribution, and campaign reporting.” Then test it against “Fraud campaign generates synthetic clicks”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invarianteach accepted advertisement serving operation has one stable result and retries do not repeat its business effect.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects each accepted advertisement serving operation has one stable result and retries do not repeat its business effect. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top