29 / 60 · Platform

Multi-Tenant SaaS — Production Architecture

All designs

Isolate tenants across identity, data, compute, quotas, and operations.

100K tenants · enterprise isolation tiers · global regions
1. High-level architecture and animated call flow

Choose a path, then play or step through exactly how the call moves between components.

← / → step · Space play
CLOUD / MULTI-AZ DEPLOYMENT MULTI-TENANT SAAS · PRODUCTION TOPOLOGY 15 named components · solid = request/data dependency · warm curve = control, event, or repair path MULTI-ZONE · VERSIONED · REPLAYABLE 01 · REQUEST & CONTROL02 · DOMAIN & COORDINATION03 · DATA, ASYNC & OPERATIONSrequestauthenticateroutevalidateread / reservecommitpublishconsumeprojectobserverepairreconcileinvalidateacknowledgepolicy snapshotevent streamtelemetry 01 SERVICE Tenant User interaction boundary 02 SERVICE Identity Provider domain subsystem 03 EDGE Global Gateway protocol · auth · quota 04 SERVICE Tenant Context domain subsystem 05 SERVICE Quota/Entitlement domain subsystem 06 EDGE Tenant Router routing & ownership 07 SERVICE Shared ServiceFleet domain subsystem 08 SERVICE Placement Directory routing & ownership 09 DATA Pooled DB Shards authoritative durable state 10 DATA Dedicated DB Tier authoritative durable state 11 SERVICE Per-Tenant KeyService domain subsystem 12 STREAM Usage Event Stream durable ordered events 13 SERVICE Billing/Metering domain subsystem 14 SERVICE Support AccessBroker domain subsystem 15 OPS Isolation Monitor evidence & operations externaledgeservicedatastreamsecurityops
Step 1 / 7Global Gateway

The caller starts the operation at Global Gateway with a stable request and retry identity.

  1. 01 · Global GatewayThe caller starts the operation at Global Gateway with a stable request and retry identity.
  2. 02 · Tenant ContextAuthenticate identity and derive tenant context from trusted claims, never request payload.
  3. 03 · Tenant RouterResolve entitlements, quotas, encryption key reference, and current placement generation.
  4. 04 · Pooled DB ShardsRoute to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.
  5. 05 · Dedicated DB TierExecute with tenant ID in every primary/index key and enforce dependency budgets.
  6. 06 · Usage Event StreamEmit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.
  7. 07 · Isolation MonitorIsolation Monitor observes completion and proves the business invariant after commit.
2. Critical request and recovery sequence
MUTATION / COMMITTenant UserACTOR 01IdentityProviderACTOR 02Tenant ContextACTOR 03PlacementDirectoryACTOR 04ONLINE / READ + RECOVERYTenant UserACTOR 01Global GatewayACTOR 02Tenant RouterACTOR 03IsolationMonitorACTOR 04Authenticate identity and derive / tenant context from trusted claims,Resolve entitlements, quotas, / encryption key reference, andRoute to / pooled/partitioned/dedicated datacommitted result + eventRoute to / pooled/partitioned/dedicated dataExecute with tenant ID in every / primary/index key and enforceEmit immutable usage and isolation / signals; migrationcached/read result + cursortimeout → retry same identity → read committed truth → reconcile

Tenant User

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Identity Provider

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Global Gateway

  • Owns authentication, validation, quotas, deadlines, and routing.
  • Stateless or lease-scoped; durable progress lives elsewhere.

Tenant Context

  • Owns one bounded domain decision, transition, or operational capability.
  • Partitioned by aggregate/key and guarded by version, uniqueness, or fencing.

Complete System Design Tutorial

The design separates the latency-sensitive online path, the correctness-critical durable transition, and asynchronous projection or repair work. Each chapter below explains one decision shown in the diagrams.

1. Requirements and scope

Functional requirements

  • Execute the primary multi-tenant saas workflow and return an observable result.
  • Expose the concrete operations shown below with stable retry identities.
  • Allow operators to inspect, disable, repair, or replay ambiguous state.
  • Keep derived analytics and notifications off the correctness-critical path.

Non-functional requirements

  • Design for 100K tenants · enterprise isolation tiers · global regions.
  • Protect the named business invariant under concurrency and partial failure.
  • Scale by an explicit aggregate/partition key without one global mutex.
  • Degrade intentionally when cache, queue, store, or regional ownership is unavailable.
Primary invariant: each accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect.
↑ top

2. Capacity assumptions

State assumptions before choosing technology. The supplied workload is 100K tenants · enterprise isolation tiers · global regions. Convert it into peak throughput, retained records, bandwidth, and failure budget; then connect each number to a component.

DimensionWorking estimateDesign implication
Traffic100K tenants · enterprise isolation tiers · global regionsSeparate online compute, durable state, and asynchronous consumers.
Peak factor5–10× averageAdmission control, autoscaling headroom, and hot-key protection.
Stored statewrite rate × retention × record bytesPartition count, indexes, compaction, backup, and archival tier.
Networkpeak operations × payload sizeBatching, compression, edge placement, and egress budget.
Availability99.9–99.99%Multi-zone replicas, tested failover, and dependency error budgets.
RecoveryRPO/RTO by invariantAck boundary, replay retention, reconciliation, and operator runbooks.
Do not cargo-cult the numbers: show the arithmetic, name the uncertain variable, and say which production metric will replace the interview estimate.
↑ top

3. API and data model

POST /v1/tenantsprovision placement/policy
GET /v1/resources/{id}trusted tenant scope
POST /v1/tenants/{id}/migrationsstaged move
GET /v1/usagemetered dimensions

Authoritative records and indexes

tenant(tenant_id PK, tier, placement_id, key_ref, policy_version)

placement(placement_id, region, mode, shard_or_database, generation)

resource(tenant_id, resource_id, version, payload)

usage_event(event_id, tenant_id, dimension, quantity, occurred_at)

Command and event envelopes

# Command envelope
operation: POST /v1/tenants
request_id: req_01J...
idempotency_key: idem_7f2...
aggregate_key: multi-tenant-saas_123
expected_version: 17
payload: {...}

# Durable event
event_id: evt_01J...
aggregate_key: multi-tenant-saas_123
event_type: multi_tenant_saas_committed
schema_version: 3
trace_id: tr_8ac...

The authoritative aggregate and idempotency record share a transactional or conditional-write boundary. Event streams, search indexes, caches, analytics, and dashboards are rebuildable projections.

↑ top

4. Why this architecture and what it costs

The boxes are not a shopping list. Each boundary exists because it owns a different invariant, scaling dimension, data lifecycle, security policy, or failure domain. The table makes those choices explicit and shows when the design should remain simpler.

ComponentOwned responsibilityWhy separate itRejected alternativeCost and evolution trigger
Tenant UserOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate identity and derive tenant context from trusted claims, never request payload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents the external caller from also owning the contract and failure behavior of Identity Provider.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Missing tenant predicate leaks data” to the external caller.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Tenant User boundary before extracting or scaling it.
Identity ProviderOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Resolve entitlements, quotas, encryption key reference, and current placement generation.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Tenant User from also owning the contract and failure behavior of Global Gateway.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Noisy tenant exhausts shared pool” to Tenant User.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Identity Provider boundary before extracting or scaling it.
Global GatewayOwns authentication, validation, quotas, deadlines, and routing. In this design it participates in: Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Identity Provider from also owning the contract and failure behavior of Tenant Context.Put protocol, authentication, quotas, and business transitions in one process. Acceptable at low scale, but it couples external traffic risk to domain releases. Here that simpler choice would couple “Placement migration reads two versions” to Identity Provider.Costs another hop, policy distribution, and a high-blast-radius fleet. Introduce it when many clients or services need one protocol and protection boundary. Measure the latency, saturation, lag, or mismatch at the Global Gateway boundary before extracting or scaling it.
Tenant ContextOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Execute with tenant ID in every primary/index key and enforce dependency budgets.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Global Gateway from also owning the contract and failure behavior of Quota / Entitlement.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Support impersonation bypasses audit” to Global Gateway.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Tenant Context boundary before extracting or scaling it.
Quota / EntitlementOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Emit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Tenant Context from also owning the contract and failure behavior of Tenant Router.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Per-tenant key service unavailable” to Tenant Context.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Quota / Entitlement boundary before extracting or scaling it.
Tenant RouterOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate identity and derive tenant context from trusted claims, never request payload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Quota / Entitlement from also owning the contract and failure behavior of Shared Service Fleet.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Missing tenant predicate leaks data” to Quota / Entitlement.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Tenant Router boundary before extracting or scaling it.
Shared Service FleetOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Resolve entitlements, quotas, encryption key reference, and current placement generation.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Tenant Router from also owning the contract and failure behavior of Placement Directory.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Noisy tenant exhausts shared pool” to Tenant Router.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Shared Service Fleet boundary before extracting or scaling it.
Placement DirectoryOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Shared Service Fleet from also owning the contract and failure behavior of Pooled DB Shards.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Placement migration reads two versions” to Shared Service Fleet.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Placement Directory boundary before extracting or scaling it.
Pooled DB ShardsOwns authoritative versioned state or immutable evidence. In this design it participates in: Execute with tenant ID in every primary/index key and enforce dependency budgets.Defines the source of truth and commit boundary. It prevents Placement Directory from also owning the contract and failure behavior of Dedicated DB Tier.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Support impersonation bypasses audit” to Placement Directory.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Pooled DB Shards boundary before extracting or scaling it.
Dedicated DB TierOwns authoritative versioned state or immutable evidence. In this design it participates in: Emit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.Defines the source of truth and commit boundary. It prevents Pooled DB Shards from also owning the contract and failure behavior of Per-Tenant Key Service.Let each service persist its own interpretation. That removes one clear commit boundary and makes retries, support, and reconciliation ambiguous. Here that simpler choice would couple “Per-tenant key service unavailable” to Pooled DB Shards.Costs schema migration, replication, backup, and shard ownership. Split storage only when access patterns, retention, or compliance boundaries materially differ. Measure the latency, saturation, lag, or mismatch at the Dedicated DB Tier boundary before extracting or scaling it.
Per-Tenant Key ServiceOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Authenticate identity and derive tenant context from trusted claims, never request payload.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Dedicated DB Tier from also owning the contract and failure behavior of Usage Event Stream.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Missing tenant predicate leaks data” to Dedicated DB Tier.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Per-Tenant Key Service boundary before extracting or scaling it.
Usage Event StreamPreserves durable ordering, replay, and consumer checkpoints. In this design it participates in: Resolve entitlements, quotas, encryption key reference, and current placement generation.Decouples bursts and failure from user acknowledgment. It prevents Per-Tenant Key Service from also owning the contract and failure behavior of Billing / Metering.Call every downstream consumer synchronously. That couples user latency and availability to analytics, notification, indexing, and audit dependencies. Here that simpler choice would couple “Noisy tenant exhausts shared pool” to Per-Tenant Key Service.Costs eventual consistency, schema evolution, replay tooling, idempotent consumers, and lag operations. Earn it when bursts or downstream failure must not block acknowledgment. Measure the latency, saturation, lag, or mismatch at the Usage Event Stream boundary before extracting or scaling it.
Billing / MeteringOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Usage Event Stream from also owning the contract and failure behavior of Support Access Broker.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Placement migration reads two versions” to Usage Event Stream.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Billing / Metering boundary before extracting or scaling it.
Support Access BrokerOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Execute with tenant ID in every primary/index key and enforce dependency budgets.Separates a distinct scale, ownership, policy, or failure boundary. It prevents Billing / Metering from also owning the contract and failure behavior of Isolation Monitor.Keep this behavior inside an adjacent service. Prefer that simpler option until independent scale, ownership, policy, or failure isolation is measurable. Here that simpler choice would couple “Support impersonation bypasses audit” to Billing / Metering.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Support Access Broker boundary before extracting or scaling it.
Isolation MonitorOwns one bounded domain decision, transition, or operational capability. In this design it participates in: Emit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.Proves correctness and supports controlled recovery. It prevents Support Access Broker from also owning the contract and failure behavior of operators and repair tooling.Infer correctness from request logs. Logs do not prove durable business state, detect silent drift, or safely repair ambiguous outcomes. Here that simpler choice would couple “Per-tenant key service unavailable” to Support Access Broker.Costs an interface, deployment, telemetry, ownership, and failure mode. Keep it in-process first; extract when independent scale or reliability pays for the boundary. Measure the latency, saturation, lag, or mismatch at the Isolation Monitor boundary before extracting or scaling it.
Decision 01

Chosen boundary vs simpler monolith

Carry tenant identity in authenticated context, never user-controlled parameters; authorize every resource access.

A single deployable with one transactional store is the right starting point for a small multi-tenant saas workload. The proposed boundary becomes justified when carry tenant identity in authenticated context, never user-controlled parameters; authorize every resource access. Separating it contains the named failure “Missing tenant predicate leaks data” without placing unrelated work on the critical path.

Decision 02

Correctness vs availability

Choose pooled, partitioned, or dedicated data placement by compliance and noisy-neighbor requirements.

The design coordinates only where each accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect. Everywhere else it accepts bounded staleness or asynchronous propagation. The alternative—strong coordination for every projection—simplifies the mental model but increases tail latency and turns partitions into broad write unavailability.

Decision 03

Operational cost and evolution

Enforce quotas at ingress and expensive dependencies; meter usage with an immutable event trail.

This choice adds dashboards, ownership, schema compatibility, capacity planning, and recovery runbooks. Start with the minimum topology that supports “Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.” and extract another service only when measured throughput, blast radius, data lifecycle, or team ownership changes independently.

Why this design: the synchronous path contains only the decisions required to protect each accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect. Durable projections, notifications, indexing, analytics, and repair are isolated so their load or failure cannot corrupt the acknowledged operation.
Correctness boundary

Carry tenant identity in authenticated context, never user-controlled parameters; authorize every resource access.

Scale boundary

Choose pooled, partitioned, or dedicated data placement by compliance and noisy-neighbor requirements.

Operational boundary

Enforce quotas at ingress and expensive dependencies; meter usage with an immutable event trail.

↑ top

5. Critical flow and algorithm

01

Authenticate identity and derive tenant context from trusted claims, never request payload.

02

Resolve entitlements, quotas, encryption key reference, and current placement generation.

03

Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.

04

Execute with tenant ID in every primary/index key and enforce dependency budgets.

05

Emit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.

Retry-safe mutation skeleton

identity = authenticate(request)
prior = idempotency.get(identity, request.key)
if prior: return prior.result

current = authoritative_store.read(partition_key)
next_state, events = domain.transition(current, request)
authoritative_store.compare_and_set(current.version, next_state)
idempotency.commit(request.key, next_state.result)
outbox.append(events)
return next_state.result
Exactly-once business effect: the network can duplicate requests. Stable identities, conditional state transitions, immutable event IDs, and reconciliation make the user-visible effect happen once.
↑ top

6. Caching, hot keys, and backpressure

Cache policy

Add a cache only for a measured repeated read; do not place mutable correctness state in a disposable layer.

Hot partition or tenant

Measure per-key and per-tenant skew. Replicate safe reads, isolate the offender, salt only when ordering permits, and enforce budgets.

Backpressure

Bound queues, buffers, connection pools, and concurrency. Shed low-priority work and scale from lag/oldest-item age rather than CPU alone.

Stampede and retry storms

Use single-flight/coalescing, jittered expiry, bounded retries, exponential backoff, and circuit breakers so one dependency failure does not multiply traffic.

Cache outage rule: never allow every miss to fall through simultaneously. Use local protection, request coalescing, circuit breakers, bounded pools, and controlled fallback.
↑ top

7. Partitioning and consistency

partition = stable_hash(primary_or_aggregate_key) → shard owner
write = conditional_update(expected_version)
read_projection = versioned and rebuildable

Strong boundary

each accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect. Use a uniqueness constraint, serializable/conditional write, expected version, or fencing token at this boundary.

Eventual boundary

Analytics, notifications, search, read models, and operational dashboards may lag when freshness is observable and replay exists.

OperationConsistencyReason
Authoritative mutationConditional / serializableProtect uniqueness, ownership, money, inventory, order, or legal state transition.
Read after own writePrimary, session token, or cache warmThe caller must observe the result it just committed.
Ordinary read projectionBounded stale where safeReplicas and caches improve latency when version/freshness is visible.
Analytics and operationsEventual + replayFreshness is less important than isolating the online path.
  • Route every mutation for one aggregate to one owner/partition epoch.
  • Use cursors rather than offsets for changing ordered collections.
  • Retain tombstones until replication and repair windows close.
  • Move shard ownership through versioned maps and dual-read verification.
↑ top

8. Failure drills and recovery

FailureRuntime behaviorRecovery proof
Missing tenant predicate leaks dataStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Noisy tenant exhausts shared poolStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Placement migration reads two versionsResolve at source of truth with uniqueness, expected version, lease generation, or monotonic cursor.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Support impersonation bypasses auditStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.
Per-tenant key service unavailableStop unsafe progress, serve only explicitly safe degraded behavior, and recover from durable state.Reconciliation returns zero invariant gaps; lag/replica health and user-visible SLO return to baseline.

Recovery checkpoint

# Recovery checkpoint
scenario: Missing tenant predicate leaks data
operation_id: op_01J...
aggregate_key: multi-tenant-saas_123
last_durable_version: 18
projection_checkpoint: partition-07@offset-482901
retry_policy: same_identity / bounded_backoff
reconciliation_status: pending
owner: Isolation Monitor

Every dependency needs a timeout, retry budget, circuit policy, and explicit fail-open, fail-closed, stale, or read-only mode. “Highly available” is incomplete until degraded behavior is named.

↑ top

9. Security, regions, and observability

Security and abuse

  • Authenticate at ingress and authorize the exact resource/tenant on every operation.
  • Encrypt transport and durable data; tokenize or isolate sensitive material.
  • Rate-limit by tenant, actor, resource, and network; detect enumeration and anomalous fan-out.
  • Redact secrets and personal data before logs or analytical storage.
  • Audit privileged mutations, support access, policy changes, and forced recovery.

Signals that prove correctness

  • User-visible success rate and p50/p95/p99 latency.
  • Timeouts, circuit state, saturation, and replica health by dependency.
  • Queue lag, oldest-item age, retries, and dead-letter growth.
  • Conflict, duplicate-suppression, and reconciliation mismatch counts.
  • Business drill: Missing tenant predicate leaks data — alert before the user-visible invariant is violated.

Regional evolution

  1. Begin multi-zone with tested failover, backups, and reconciliation.
  2. Add global stateless ingress and regional read projections.
  3. Keep one write owner per aggregate or partition where coordination is expensive.
  4. Move ownership with epochs/fencing; avoid silent simultaneous writers.
  5. Use expand/contract schemas, canaries, shadow reads, and reversible cutovers.
↑ top

10. Interviewer follow-ups

When does a tenant get a dedicated database?

Carry tenant identity in authenticated context, never user-controlled parameters; authorize every resource access. Walk the concrete lifecycle step “Authenticate identity and derive tenant context from trusted claims, never request payload.” Then test it against “Missing tenant predicate leaks data”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you prevent cross-tenant leaks?

Choose pooled, partitioned, or dedicated data placement by compliance and noisy-neighbor requirements. Walk the concrete lifecycle step “Resolve entitlements, quotas, encryption key reference, and current placement generation.” Then test it against “Noisy tenant exhausts shared pool”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do per-tenant encryption keys work?

Enforce quotas at ingress and expensive dependencies; meter usage with an immutable event trail. Walk the concrete lifecycle step “Route to pooled/partitioned/dedicated data using tenant-scoped credentials and keys.” Then test it against “Placement migration reads two versions”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you migrate a tenant?

Carry tenant identity in authenticated context, never user-controlled parameters; authorize every resource access. Walk the concrete lifecycle step “Execute with tenant ID in every primary/index key and enforce dependency budgets.” Then test it against “Support impersonation bypasses audit”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

How do you test isolation?

Choose pooled, partitioned, or dedicated data placement by compliance and noisy-neighbor requirements. Walk the concrete lifecycle step “Emit immutable usage and isolation signals; migration dual-reads/checks before pointer cutover.” Then test it against “Per-tenant key service unavailable”. State the retry identity, durable checkpoint, recovery signal, and cost of the alternative.

Answer pattern: state the invariant, choose a mechanism, walk one failure, name the recovery signal, and explain the cost of the alternative.
↑ top

11. How to present this in 45 minutes

TimeWhat to cover
0–5 minClarify scope, workload, SLOs, and the correctness invariant.
5–10 minEstimate throughput/storage and define concrete APIs and records.
10–20 minDraw online, durable-write, and asynchronous/repair paths.
20–30 minExplain the exact lifecycle, partition key, and concurrency control.
30–38 minDiscuss caching, hot keys, replication, and backpressure.
38–43 minRun failure drills, security, observability, and regional evolution.
43–45 minSummarize the five most important choices and trade-offs.

Final decision summary

Invarianteach accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect.
PartitioningStable aggregate/key hash with versioned ownership and measured hot-key isolation.
CommitExpected version or uniqueness at the authoritative state boundary.
Async workDurable outbox/log, idempotent consumers, checkpoints, and replay.
RecoverySame operation identity, durable truth lookup, and business reconciliation.

Closing summary

CLOSING “The design protects each accepted multi-tenant saas operation has one stable result and retries do not repeat its business effect. The online path stays bounded, authoritative state is versioned and partitioned, and asynchronous work is replayable and observable.”
↑ top