14 / 60 · Data Platforms

Web Crawler — Production Architecture

All designs

Fetch the web politely, deduplicate URLs, and scale parsing and storage independently.

10B URLs · 100K fetches/s · petabytes of content
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT WEB CRAWLER · PRODUCTION TOPOLOGY 14 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidatepolicy snapshotevent streamtelemetry 01 SERVICE Seeds/Sitemaps domain subsystem 02 SERVICE URL Normalizer domain subsystem 03 SERVICE Seen Bloom Filter domain subsystem 04 SERVICE Durable URL Set domain subsystem 05 SERVICE Host Frontier domain subsystem 06 SERVICE PolitenessScheduler domain subsystem 07 EDGE DNS Cache low-latency read path 08 SERVICE Fetcher Fleet domain subsystem 09 SERVICE Robots Cache low-latency read path 10 DATA Content Store authoritative durable state 11 SERVICE Parser/Extractor asynchronous execution 12 SERVICE Canonical Resolver domain subsystem 13 SERVICE Link Graph domain subsystem 14 SERVICE Recrawl Scorer domain subsystem externaledgeservicedatastreamsecurityops
Step 1 / 6Seeds / Sitemaps

The caller starts the operation at Seeds / Sitemaps with a stable request and retry identity.

  1. 01 · Seeds / SitemapsThe caller starts the operation at Seeds / Sitemaps with a stable request and retry identity.
  2. 02 · URL NormalizerNormalize and canonicalize a discovered URL before checking fast and durable dedupe.
  3. 03 · Seen Bloom FilterPlace it in a per-host frontier ordered by priority and next allowed fetch time.
  4. 04 · Durable URL SetA politeness scheduler leases eligible URLs while enforcing robots and host concurrency.
  5. 05 · Fetcher FleetFetch with strict size/type/time limits, persist raw content, parse links, and commit result.
  6. 06 · Recrawl ScorerScore change frequency and importance to schedule the next crawl.
2. Critical request and recovery sequence
MUTATION / COMMITSeeds/SitemapsACTOR 01URL NormalizerACTOR 02Durable URLSetACTOR 03Fetcher FleetACTOR 04ONLINE / READ + RECOVERYSeeds/SitemapsACTOR 01Seen BloomFilterACTOR 02PolitenessSchedulerACTOR 03Recrawl ScorerACTOR 04Normalize and canonicalize a / discovered URL before checking fastPlace it in a per-host frontier / ordered by priority and nextA politeness scheduler leases / eligible URLs while enforcingcommitted result + eventA politeness scheduler leases / eligible URLs while enforcingFetch with strict size/type/time / limits, persist raw content, parseScore change frequency and / importance to schedule the nextcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

Seeds / Sitemaps

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

URL Normalizer

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Seen Bloom Filter

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Durable URL Set

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary web crawler workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for 10B URLs · 100K fetches/s · petabytes of content.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: each accepted web crawler operation has one stable result and retries do not repeat its business effect.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 10B URLs · 100K fetches/s · petabytes of content. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
Traffic10B URLs · 100K fetches/s · petabytes of contentSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

POST /v1/seedsadd crawl scope
GET /v1/frontier/statshost backlog
PUT /internal/fetch-resultcommit outcome and links
POST /v1/recrawladjust freshness policy

Authoritative records and indexes

url(url_hash PK, canonical_url, host, state, next_fetch_at)

host(host_hash, robots_version, delay_ms, next_allowed_at)

fetch(url_hash, fetched_at, status, content_hash, storage_uri)

link(source_hash, target_hash, discovered_at)

Command and event envelopes

# Command envelope
operation: POST /v1/seeds
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: web-crawler_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: web-crawler_123
event_type: web_crawler_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
Seeds / SitemapsOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Normalize and canonicalize a discovered URL before checking fast and durable dedupe.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of URL Normalizer.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Crawler trap generates infinite URL space” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Seeds / Sitemaps boundary before extracting or scaling it.
URL NormalizerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Place it in a per-host frontier ordered by priority and next allowed fetch time.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Seeds / Sitemaps from also owning the contract and failure behavior of Seen Bloom Filter.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Slow host holds worker resources” to Seeds / Sitemaps.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the URL Normalizer boundary before extracting or scaling it.
Seen Bloom FilterOwns one bounded domain decision, transition, or operational capability. In this design it participates in: A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.Separates a distinct scale, ownership, policy, or failure boundary. It prevents URL Normalizer from also owning the contract and failure behavior of Durable URL Set.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “robots.txt changes during crawl” to URL Normalizer.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Seen Bloom Filter boundary before extracting or scaling it.
Durable URL SetOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Fetch with strict size/type/time limits, persist raw content, parse links, and commit result.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Seen Bloom Filter from also owning the contract and failure behavior of Host Frontier.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Duplicate content has many URLs” to Seen Bloom Filter.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Durable URL Set boundary before extracting or scaling it.
Host FrontierOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Score change frequency and importance to schedule the next crawl.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Durable URL Set from also owning the contract and failure behavior of Politeness Scheduler.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “JavaScript renderer exhausts sandbox” to Durable URL Set.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Host Frontier boundary before extracting or scaling it.
Politeness SchedulerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Normalize and canonicalize a discovered URL before checking fast and durable dedupe.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Host Frontier from also owning the contract and failure behavior of DNS Cache.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Crawler trap generates infinite URL space” to Host Frontier.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Politeness Scheduler boundary before extracting or scaling it.
DNS CacheServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Place it in a per-host frontier ordered by priority and next allowed fetch time.Removes repeated work from the latency-critical path. It prevents Politeness Scheduler from also owning the contract and failure behavior of Fetcher Fleet.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Slow host holds worker resources” to Politeness Scheduler.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (10B URLs · 100K fetches/s · petabytes of content) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the DNS Cache boundary before extracting or scaling it.
Fetcher FleetOwns one bounded domain decision, transition, or operational capability. In this design it participates in: A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.Separates a distinct scale, ownership, policy, or failure boundary. It prevents DNS Cache from also owning the contract and failure behavior of Robots Cache.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “robots.txt changes during crawl” to DNS Cache.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Fetcher Fleet boundary before extracting or scaling it.
Robots CacheServes a low-latency, disposable projection with authoritative fallback. In this design it participates in: Fetch with strict size/type/time limits, persist raw content, parse links, and commit result.Removes repeated work from the latency-critical path. It prevents Fetcher Fleet from also owning the contract and failure behavior of Content Store.Read authoritative storage for every request. Simpler initially, but latency and source-of-truth load scale together and a hot key becomes a database incident. Here that simpler choice would couple “Duplicate content has many URLs” to Fetcher Fleet.Costs invalidation, staleness, stampede protection, and memory. Add or split it only after the workload (10B URLs · 100K fetches/s · petabytes of content) proves repeated reads dominate. Measure the latency, saturation, lag, or mismatch at the Robots Cache boundary before extracting or scaling it.
Content StoreOwns authoritative versioned state or immutable evidence. In this design it participates in: Score change frequency and importance to schedule the next crawl.Defines the source of truth and commit boundary. It prevents Robots Cache from also owning the contract and failure behavior of Parser / Extractor.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “JavaScript renderer exhausts sandbox” to Robots Cache.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Content Store boundary before extracting or scaling it.
Parser / ExtractorOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Normalize and canonicalize a discovered URL before checking fast and durable dedupe.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Content Store from also owning the contract and failure behavior of Canonical Resolver.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Crawler trap generates infinite URL space” to Content Store.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Parser / Extractor boundary before extracting or scaling it.
Canonical ResolverOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Place it in a per-host frontier ordered by priority and next allowed fetch time.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Parser / Extractor from also owning the contract and failure behavior of Link Graph.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Slow host holds worker resources” to Parser / Extractor.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Canonical Resolver boundary before extracting or scaling it.
Link GraphOwns one bounded domain decision, transition, or operational capability. In this design it participates in: A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Canonical Resolver from also owning the contract and failure behavior of Recrawl Scorer.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “robots.txt changes during crawl” to Canonical Resolver.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Link Graph boundary before extracting or scaling it.
Recrawl ScorerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Fetch with strict size/type/time limits, persist raw content, parse links, and commit result.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Link Graph from also owning the contract and failure behavior of operators and repair tooling.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Duplicate content has many URLs” to Link Graph.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Recrawl Scorer boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Partition the frontier by host to enforce robots.txt, crawl delay, and per-domain fairness.

A single deployable with one transactional store is the right starting point for a small web crawler workload. The proposed boundary becomes justified when partition the frontier by host to enforce robots.txt, crawl delay, and per-domain fairness. Separating it contains the named failure “Crawler trap generates infinite URL space” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Canonicalize before deduplication; use Bloom filters as a fast first check and durable hashes as truth.

The design coordinates only where each accepted web crawler operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Separate fetch, parse, and indexing stages with queues so slow domains cannot stall the fleet.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect each accepted web crawler operation has one stable result and retries do not repeat its business effect. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Partition the frontier by host to enforce robots.txt, crawl delay, and per-domain fairness.

Scale boundary

Canonicalize before deduplication; use Bloom filters as a fast first check and durable hashes as truth.

Operational boundary

Separate fetch, parse, and indexing stages with queues so slow domains cannot stall the fleet.

↑ top

5. Critical flow and algorithm

01

Normalize and canonicalize a discovered URL before checking fast and durable dedupe.

02

Place it in a per-host frontier ordered by priority and next allowed fetch time.

03

A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.

04

Fetch with strict size/type/time limits, persist raw content, parse links, and commit result.

05

Score change frequency and importance to schedule the next crawl.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Cache immutable or slowly changing reads with versioned keys, TTL jitter, short negative entries, and request coalescing.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

each accepted web crawler operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Crawler trap generates infinite URL spaceStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Slow host holds worker resourcesStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
robots.txt changes during crawlStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Duplicate content has many URLsResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
JavaScript renderer exhausts sandboxStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Crawler trap generates infinite URL space
operation_id: op_01J...
aggregate_key: web-crawler_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Recrawl Scorer

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Crawler trap generates infinite URL space — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

How do you recrawl changed pages?

Partition the frontier by host to enforce robots.txt, crawl delay, and per-domain fairness. Walk the concrete lifecycle step “Normalize and canonicalize a discovered URL before checking fast and durable dedupe.” Then test it against “Crawler trap generates infinite URL space”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you detect crawler traps?

Canonicalize before deduplication; use Bloom filters as a fast first check and durable hashes as truth. Walk the concrete lifecycle step “Place it in a per-host frontier ordered by priority and next allowed fetch time.” Then test it against “Slow host holds worker resources”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you prioritize important URLs?

Separate fetch, parse, and indexing stages with queues so slow domains cannot stall the fleet. Walk the concrete lifecycle step “A politeness scheduler leases eligible URLs while enforcing robots and host concurrency.” Then test it against “robots.txt changes during crawl”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How is robots.txt cached?

Partition the frontier by host to enforce robots.txt, crawl delay, and per-domain fairness. Walk the concrete lifecycle step “Fetch with strict size/type/time limits, persist raw content, parse links, and commit result.” Then test it against “Duplicate content has many URLs”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you crawl JavaScript-heavy pages?

Canonicalize before deduplication; use Bloom filters as a fast first check and durable hashes as truth. Walk the concrete lifecycle step “Score change frequency and importance to schedule the next crawl.” Then test it against “JavaScript renderer exhausts sandbox”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invarianteach accepted web crawler operation has one stable result and retries do not repeat its business effect.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects each accepted web crawler operation has one stable result and retries do not repeat its business effect. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top