Designing a payment system is fundamentally an exercise in distributed correctness under partial failure rather than raw throughput scaling. At marketplace scale (50M payments/day, ~580 RPS average, 5,800 RPS peak), a small cluster of sharded SQL databases easily absorbs the write volume, yet a single dropped HTTP response, race condition, or dual-write crash directly creates or destroys real money. L4-L6 interviewers evaluate how rigorously you enforce end-to-end idempotency across external Payment Service Providers (PSPs), maintain immutable accounting invariants via double-entry bookkeeping, decouple slow external bank calls (~2 s) from internal transactions, and close external state drift through automated daily reconciliation.
WHAT YOU WILL MASTER IN THIS LESSON
Distinguish real-time authorization holds from capture and batched T+1/T+2 clearing and settlement across PSPs, acquiring banks, card networks, and issuing banks.
Guarantee exactly-once financial effects over at-least-once networks by propagating client-generated Idempotency-Key tokens through unique database constraints to external PSP endpoints.
Recover safely from ambiguous PSP timeouts and external drift using exponential-backoff retry queues, signed webhooks, dead-letter queues, and nightly settlement reconciliation.
Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.
0 / 8 Checked
RRequirements & ScopeMin 2–8
Clarify who the users are, 3–4 core functional flows, explicit non-goals, and target latency/availability SLAs before drawing a single box.
Avoid trap: Still gathering requirements at minute 15 squeezes the deep dive where L5/L6 offers are decided.
EBack-of-the-Envelope MathMin 8–12
Compute ONLY numbers that actively change architectural choices: DAU → peak read/write RPS, daily & 5-year storage, bandwidth (or tokens/sec & KV-cache VRAM for AI).
Avoid trap: Spending 10 minutes multiplying irrelevant byte counts that never influence sharding or caching.
SStorage & Data ModelMin 12–15
Define core tables, primary/partition keys, and access indexes. Separate small structured metadata rows (~1 KB) from bulk binary blobs (S3 + CDN).
Avoid trap: Picking Cassandra or NoSQL by buzzword when 40 writes/s & 3 TB fit cleanly on one indexed SQL primary + replicas.
AAPIs & Wire ContractsMin 15–17
Write 3–4 concrete REST/gRPC/SSE/WebSocket endpoints with request/response shapes, cursor pagination, and Idempotency-Key headers on mutations.
Avoid trap: Hand-waving client-server contracts and discovering missing parameters halfway through the deep dive.
HHigh-Level DesignMin 17–27
Draw the simplest end-to-end working read and write paths first. Every box on the board must trace back to a stated requirement or number.
Avoid trap: Reciting a memorized 12-microservice Kafka mesh before proving why a simpler synchronous path breaks.
DDetailed Design (Deep Dive)Min 27–36
Proactively audit your high-level diagram, name the top 2 bottlenecks (hot partitions, fan-out skew, lock contention, GPU saturation), and go deep on internals.
Avoid trap: Waiting passively for the interviewer to rescue pacing or skimming every box at shallow surface depth.
EEvaluation & Failure ModesMin 36–41
State what each architectural choice COSTS you, and trace graceful degradation when cache, queue, external vendor, or primary DB fails.
Avoid trap: Assuming happy-path uptime or placing slow 3-second external calls inline on revenue-critical write paths.
DDistinctive FeatureMin 41–45
Solve the signature hard problem unique to this prompt (e.g., double-entry ledger idempotency, Geohash dispatch locks, OT/CRDT sync, or SSE token streaming).
Avoid trap: Spending 20 minutes on generic login/CRUD tables and zero time on the core distributed challenge.
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?
5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)
Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL
Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.
✗ NO-HIRE RED FLAG
Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.
2. Right-Sized Design
✓ STRONG HIRE SIGNAL
Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).
✗ NO-HIRE RED FLAG
Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.
3. Technical Depth
✓ STRONG HIRE SIGNAL
Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.
✗ NO-HIRE RED FLAG
Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.
4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL
Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'
✗ NO-HIRE RED FLAG
Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.
5. Communication
✓ STRONG HIRE SIGNAL
Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.
✗ NO-HIRE RED FLAG
Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.
ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 6Section 01: Introduction to the Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
SECTION 01 OF 2
1. Introduction to the Payment System
Key Takeaway: Payment throughput is modest (~580 RPS average, 5,800 RPS peak), so architecture must prioritize zero-error financial consistency, non-blocking external I/O across 11,600 concurrent in-flight PSP calls, and PCI-DSS scope isolation before horizontal write scaling.
SECTION 01 DIAGRAMS · SLIDE 1 OF 2Introduction to the Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
Payment Rails: Authorization vs. Clearing and Settlement
When a buyer clicks Pay on a $50.00 order and sees approval within 2 s, no actual money has moved between banks yet. An online merchant never connects directly to banking mainframes; instead, it integrates with a Payment Service Provider (PSP) such as Stripe or Adyen, which routes requests through the Acquiring Bank (the merchant's bank) over the Card Network (Visa or Mastercard) to the Issuing Bank (the buyer's bank). The initial synchronous round-trip is Authorization, where the issuing bank evaluates balance and fraud rules and places a $50.00 hold on the cardholder's line of credit. Later, typically when physical goods ship, the merchant triggers Capture to convert the hold into a billable charge, while actual inter-bank fund transfers occur asynchronously in daily batches (T+1 or T+2) during Clearing and Settlement.
Merchant Scope Boundary: Our internal payment system orchestrates checkout state, calls the external PSP, manages seller balances, and maintains internal accounting books.
Ambiguous Timeout Hazard: If our system sends a $50.00 charge request to the PSP and the response drops in transit, blind retries risk charging $100.00 while abandoning the call risks shipping unpaid goods.
Exactly-Once Effect Requirement: Because TCP and HTTP retries provide at-least-once delivery at best, every state transition and external call must be strictly idempotent.
PCI-DSS Tokenization: Raw Primary Account Numbers (PANs) are high-value attack targets; having the browser send card details directly to the PSP in exchange for a single-use card_token keeps merchant servers out of heavy PCI-DSS compliance scope.
Interview Signal: Interviewer Tip: Clarify early whether you are designing a merchant/marketplace payment orchestrator (integrating with PSPs like Stripe/Adyen) or card-network switching infrastructure itself. Nearly all FAANG interviews target the merchant/marketplace payment platform.
Functional and Non-Functional Requirements
A production marketplace payment engine covers both inbound buyer collections (pay-ins) and outbound seller disbursements (payouts), alongside full or partial refunds modeled as reverse money flows. Unlike social feeds or search engines that trade staleness for sub-50 ms latency under the CAP theorem, financial systems invert that priority: a 2,000 ms checkout response or a temporary PENDING status is acceptable, whereas losing a payment, double-charging a buyer, or corrupting a ledger balance is unacceptable.
Category
Requirement
Engineering Implication
Functional
Accept Payments (Pay-in)
Charge cards or alternative payment methods via PSP tokens with end-to-end idempotency.
Functional
Refunds & Partial Refunds
Execute compensating reverse charges and append reversing ledger entries without mutating history.
Functional
Marketplace Payouts
Aggregate settled merchant balances and disburse seller shares on a daily or scheduled cadence.
Functional
Payment Status Visibility
Expose deterministic PENDING, SUCCEEDED, or FAILED state to buyers and merchants at all times.
Functional
Daily Reconciliation
Compare internal ledger entries line-by-line against nightly PSP and bank settlement reports.
Non-Functional
Reliability & Zero Double-Charge
Survive process crashes, duplicate user clicks, and network timeouts with exactly-once financial effect.
Non-Functional
Strong Financial Consistency
Enforce ACID transactions and sum(debits) = sum(credits) invariants across all internal accounts.
Assuming a large global marketplace processing 50,000,000 (50M) payments per day, baseline throughput is 50,000,000 / 86,400 s ≈ 580 RPS, rising 10x during flash sales or holiday peaks to ~5,800 RPS. While 5,800 writes/s fits comfortably on a modestly sharded relational database, external PSP round-trips average ~2 s because they traverse card networks and issuing-bank fraud engines. By Little's Law (L = lambda * W), 5,800 RPS * 2 s = 11,600 payment executions are simultaneously in flight at peak, meaning synchronous thread-per-request blocking would exhaust application thread pools and cascade into site-wide outages.
Throughput is modest; avoid sacrificing SQL ACID for NoSQL write scale.
Concurrent In-Flight Calls
5,800 RPS * 2 s PSP wait
11,600 open calls
Mandates non-blocking async I/O and queue decoupling before PSP calls.
Daily Storage Growth
50M * 2 KB (payment + ledger)
100 GB/day
Compact per-transaction footprint including double-entry rows.
7-Year Audit Retention
100 GB/day * 365 * 7
255.5 TB
Partition historical ledger partitions by time and tier cold years to archival storage.
Interview Signal: Common Candidate Mistake: Reaching for Cassandra or DynamoDB because '50 million payments a day sounds huge.' At 5,800 RPS peak, sharded PostgreSQL or MySQL easily absorbs the load while giving you multi-table ACID transactions that prevent catastrophic balance drift.
SECTION 02 OF 2
2. Design of a Payment System
Key Takeaway: Combine client-to-PSP idempotency keys, single-shard SQL transactions with a transactional outbox, append-only double-entry bookkeeping, and nightly PSP report reconciliation to ensure every cent is accounted for even when networks and services fail.
SECTION 02 DIAGRAMS · SLIDE 1 OF 4Design of a Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
High-Level Architecture, API, and Relational Schema
When a buyer submits checkout, the client first exchanges raw card details directly with the PSP for a card_token, then calls POST /payments carrying an Idempotency-Key UUID generated once per checkout intent. The Payment Service immediately persists the payment record with status = 'PENDING' in Payment DB before executing any external call so a subsequent crash never loses track of the order. A synchronous Risk Check service scores device, velocity, and behavioral signals to block stolen cards before incurring PSP fees or chargebacks, after which a decoupled Payment Executor invokes the PSP asynchronously using non-blocking I/O. Once the PSP responds (either inline or minutes later via an HMAC-signed POST /webhooks/psp callback for 3D-Secure/async payment methods), downstream Ledger and Wallet services record the financial movements.
Table
Core Columns
Constraints & Concurrency Control
Role in Architecture
payments
payment_id (PK), idem_key, order_id, amount_cents, currency, status
Preventing double charges requires enforcing idempotency across two distinct network hops. On the first hop (Checkout -> Payment Service), if a user double-taps Pay on a flaky mobile connection, both requests carry the identical Idempotency-Key; the first inserts a payments row while the second hits the UNIQUE(idem_key) constraint, waits for or reads the existing record, and returns the exact same result. On the second hop (Payment Executor -> PSP), we pass our internal payment_id as the PSP's idempotency key so that retrying after a 2 s socket timeout causes the PSP to return the original charge rather than billing the card twice. Once succeeded, we record funds using Double-Entry Bookkeeping rather than mutating a single balance integer: every financial event writes two or more immutable ledger rows whose debits equal credits, making any rounding bug or partial write immediately detectable as an arithmetic imbalance.
Ledger Entry
Account
Entry Type
Amount (USD)
Financial Meaning (`$50.00` Order, `3%` Fee)
Entry 1
PSP_Clearing_Receivable
Debit
$50.00
External PSP now owes the platform $50.00 upon T+1 settlement.
Entry 2
Merchant_Payable (Seller)
Credit
$48.50
Platform owes the seller $50.00 - 3% ($48.50) at next payout.
Entry 3
Platform_Fee_Revenue
Credit
$1.50
Platform recognizes $1.50 (3%) take-rate revenue.
Invariant Check
SUM(Debits) == SUM(Credits)
Balanced
$50.00 == $50.00
Zero cents created or destroyed; refunds append opposite entries.
Interview Signal: Always store monetary amounts as exact integer minor units (amount_cents BIGINT, e.g., 5000 for $50.00) paired with an ISO-4217 currency code (USD, JPY), or fixed-precision NUMERIC(19,4)—never IEEE-754 floating-point (FLOAT/DOUBLE), where 0.1 + 0.2 != 0.3 breaks ledger balance checks.
Timeout Recovery, Exponential Backoff, and the Transactional Outbox
When a PSP request times out, the executor does not know whether the issuing bank placed the hold or dropped the packet. The cardinal rule is never to mark a timed-out payment FAILED on a guess: its status remains PENDING, and the executor enqueues a retry job carrying the original payment_id idempotency key onto a dedicated Retry Queue with exponential backoff (1 s, 2 s, 4 s, 8 s plus jitter) so worker threads remain free and struggling PSPs are not hammered. If retries continue failing past a threshold while no signed webhook arrives, the job routes to a Dead Letter Queue (DLQ) for automated PSP status-API polling or human investigation. Similarly, updating payments.status = 'SUCCEEDED' and publishing events to Ledger, Wallet, and Notifications avoids the classic Dual-Write Problem via the Transactional Outbox Pattern: the SQL transaction commits both the payments update and an outbox row atomically, and an asynchronous Relay publishes outbox rows to the Event Queue (at-least-once) while downstream consumers deduplicate by payment_id.
Pending-Until-Proven Rule: Every payment terminates in SUCCEEDED, FAILED (confirmed decline), or an actively monitored DLQ—nothing is silently dropped.
Webhook Convergence: Asynchronous payment methods (3DS challenges, bank transfers) and race conditions resolve cleanly when the HMAC-verified PSP webhook settles the PENDING row directly.
Outbox Crash Safety: If the Relay crashes after publishing to Kafka/SQS but before marking outbox.sent = true, it republishes on restart; idempotent Ledger and Wallet consumers ignore already-applied payment_id events.
Consumer Isolation: Slow downstream email receipts or analytics consumers lag independently on the Event Queue without adding latency to checkout.
Nightly Settlement Reconciliation and Design Scorecard
Internal database constraints and double-entry invariants prove our own services are self-consistent, but they cannot detect bugs at the boundary with the outside banking world—such as a missed webhook, a fee-calculation discrepancy, or an unrecorded PSP refund. Every night, the PSP delivers a batch Settlement Report enumerating every captured charge, refund, and fee settled for that day. A batch Reconciliation Engine joins our internal ledger_entries against the PSP settlement report line by line (for example: p1 = $50.00 matches; p2 = $20.00 matches; p3 = $75.00 internal vs $57.00 PSP flags an amount mismatch; p4 = $12.00 internal vs missing in PSP flags a phantom pending/succeeded charge). Discrepancies route to a finance exception queue where automated rules or analysts post explicit compensating adjustment entries to close the gap.
Quality Requirement
Architectural Mechanisms
Primary Failure Mode Mitigated
Reliable
Idempotency-Key, exponential Retry Queue, DLQ
Network retries double-charging cards or PSP timeouts dropping orders.
Single-node hardware crash or slow email/PSP calls starving checkout.
Auditable
Immutable append-only ledger_entries, nightly reconciliation, 7-yr store
In-place balance tampering or silent divergence from bank settlement totals.
ACTIVE RECALL & TRADE-OFF PROBES
System Design Checkpoint Quizzes (3)
CHECKPOINT QUIZ #1
In "Introduction to the Payment System" (Functional and Non-Functional Requirements), how does a production-grade design handle "Functional" (Engineering Implication)?
CHECKPOINT QUIZ #2
In "Design of a Payment System" (High-Level Architecture, API, and Relational Schema), how does a production-grade design handle "ledger_entries" (Role in Architecture)?
CHECKPOINT QUIZ #3
Before wrapping up a 45-minute design round on "Design a Payment System", which core engineering objective should your architecture explicitly satisfy?