Dashboard/System design/41. Design a Payment System

SYSTEM DESIGN · LESSON 41 OF 52 · 13 MIN VIDEO · 2 TOPICS · ★ FREE SAMPLE VIDEO

Design a Payment System

Module Progress (0 / 2 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
Narrated HD Video Lesson · 13 min · 3 YouTube-style timestamped chapters
0:00 / 12:18
VIDEO TIMESTAMPS · 3 CHAPTERSNow at 0:00 — Module Overview
0% READSec 01/02 · Introduction to the Payment System
SYSTEM DESIGN VIDEO LESSON & GUIDE

Design a Payment System

Designing a payment system is fundamentally an exercise in distributed correctness under partial failure rather than raw throughput scaling. At marketplace scale (50M payments/day, ~580 RPS average, 5,800 RPS peak), a small cluster of sharded SQL databases easily absorbs the write volume, yet a single dropped HTTP response, race condition, or dual-write crash directly creates or destroys real money. L4-L6 interviewers evaluate how rigorously you enforce end-to-end idempotency across external Payment Service Providers (PSPs), maintain immutable accounting invariants via double-entry bookkeeping, decouple slow external bank calls (~2 s) from internal transactions, and close external state drift through automated daily reconciliation.

WHAT YOU WILL MASTER IN THIS LESSON
  • Distinguish real-time authorization holds from capture and batched T+1/T+2 clearing and settlement across PSPs, acquiring banks, card networks, and issuing banks.
  • Guarantee exactly-once financial effects over at-least-once networks by propagating client-generated Idempotency-Key tokens through unique database constraints to external PSP endpoints.
  • Design ACID relational schemas combining mutable optimistic-locked merchant wallets, immutable append-only double-entry ledgers (debits = credits), and transactional outbox tables.
  • Recover safely from ambiguous PSP timeouts and external drift using exponential-backoff retry queues, signed webhooks, dead-letter queues, and nightly settlement reconciliation.
SYSTEMATIC INTERVIEW CHECKLIST · RESHADED + 45-MIN CLOCK

8-Step RESHADED Execution Checklist

Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.

0 / 8 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 6Section 01: Introduction to the Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
$50 orderCustomerwe build thisMerchantonline shopPayment providerPSPAcquiring bankmerchant's bankCard networkhold $50Issuing bankcustomer's bank
SECTION 01 OF 2

1. Introduction to the Payment System

Key Takeaway: Payment throughput is modest (~580 RPS average, 5,800 RPS peak), so architecture must prioritize zero-error financial consistency, non-blocking external I/O across 11,600 concurrent in-flight PSP calls, and PCI-DSS scope isolation before horizontal write scaling.
SECTION 01 DIAGRAMS · SLIDE 1 OF 2Introduction to the Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
$50 orderCustomerwe build thisMerchantonline shopPayment providerPSPAcquiring bankmerchant's bankCard networkhold $50Issuing bankcustomer's bank

Payment Rails: Authorization vs. Clearing and Settlement

When a buyer clicks Pay on a $50.00 order and sees approval within 2 s, no actual money has moved between banks yet. An online merchant never connects directly to banking mainframes; instead, it integrates with a Payment Service Provider (PSP) such as Stripe or Adyen, which routes requests through the Acquiring Bank (the merchant's bank) over the Card Network (Visa or Mastercard) to the Issuing Bank (the buyer's bank). The initial synchronous round-trip is Authorization, where the issuing bank evaluates balance and fraud rules and places a $50.00 hold on the cardholder's line of credit. Later, typically when physical goods ship, the merchant triggers Capture to convert the hold into a billable charge, while actual inter-bank fund transfers occur asynchronously in daily batches (T+1 or T+2) during Clearing and Settlement.

  • Merchant Scope Boundary: Our internal payment system orchestrates checkout state, calls the external PSP, manages seller balances, and maintains internal accounting books.
  • Ambiguous Timeout Hazard: If our system sends a $50.00 charge request to the PSP and the response drops in transit, blind retries risk charging $100.00 while abandoning the call risks shipping unpaid goods.
  • Exactly-Once Effect Requirement: Because TCP and HTTP retries provide at-least-once delivery at best, every state transition and external call must be strictly idempotent.
  • PCI-DSS Tokenization: Raw Primary Account Numbers (PANs) are high-value attack targets; having the browser send card details directly to the PSP in exchange for a single-use card_token keeps merchant servers out of heavy PCI-DSS compliance scope.
Interview Signal: Interviewer Tip: Clarify early whether you are designing a merchant/marketplace payment orchestrator (integrating with PSPs like Stripe/Adyen) or card-network switching infrastructure itself. Nearly all FAANG interviews target the merchant/marketplace payment platform.

Functional and Non-Functional Requirements

A production marketplace payment engine covers both inbound buyer collections (pay-ins) and outbound seller disbursements (payouts), alongside full or partial refunds modeled as reverse money flows. Unlike social feeds or search engines that trade staleness for sub-50 ms latency under the CAP theorem, financial systems invert that priority: a 2,000 ms checkout response or a temporary PENDING status is acceptable, whereas losing a payment, double-charging a buyer, or corrupting a ledger balance is unacceptable.

CategoryRequirementEngineering Implication
FunctionalAccept Payments (Pay-in)Charge cards or alternative payment methods via PSP tokens with end-to-end idempotency.
FunctionalRefunds & Partial RefundsExecute compensating reverse charges and append reversing ledger entries without mutating history.
FunctionalMarketplace PayoutsAggregate settled merchant balances and disburse seller shares on a daily or scheduled cadence.
FunctionalPayment Status VisibilityExpose deterministic PENDING, SUCCEEDED, or FAILED state to buyers and merchants at all times.
FunctionalDaily ReconciliationCompare internal ledger entries line-by-line against nightly PSP and bank settlement reports.
Non-FunctionalReliability & Zero Double-ChargeSurvive process crashes, duplicate user clicks, and network timeouts with exactly-once financial effect.
Non-FunctionalStrong Financial ConsistencyEnforce ACID transactions and sum(debits) = sum(credits) invariants across all internal accounts.
Non-FunctionalSecurity & AuditabilityEnforce PCI-DSS card tokenization, HMAC-verified webhooks, and 7-year immutable append-only retention.

Back-of-the-Envelope Capacity Estimation

Assuming a large global marketplace processing 50,000,000 (50M) payments per day, baseline throughput is 50,000,000 / 86,400 s ≈ 580 RPS, rising 10x during flash sales or holiday peaks to ~5,800 RPS. While 5,800 writes/s fits comfortably on a modestly sharded relational database, external PSP round-trips average ~2 s because they traverse card networks and issuing-bank fraud engines. By Little's Law (L = lambda * W), 5,800 RPS * 2 s = 11,600 payment executions are simultaneously in flight at peak, meaning synchronous thread-per-request blocking would exhaust application thread pools and cascade into site-wide outages.

MetricFormulaEstimateArchitectural Consequence
Average Payment Rate50M payments / 86,400 s~580 RPSSingle-digit SQL shards handle baseline ACID writes easily.
Peak Sale Rate (10x)580 RPS * 10~5,800 RPSThroughput is modest; avoid sacrificing SQL ACID for NoSQL write scale.
Concurrent In-Flight Calls5,800 RPS * 2 s PSP wait11,600 open callsMandates non-blocking async I/O and queue decoupling before PSP calls.
Daily Storage Growth50M * 2 KB (payment + ledger)100 GB/dayCompact per-transaction footprint including double-entry rows.
7-Year Audit Retention100 GB/day * 365 * 7255.5 TBPartition historical ledger partitions by time and tier cold years to archival storage.
Interview Signal: Common Candidate Mistake: Reaching for Cassandra or DynamoDB because '50 million payments a day sounds huge.' At 5,800 RPS peak, sharded PostgreSQL or MySQL easily absorbs the load while giving you multi-table ACID transactions that prevent catastrophic balance drift.
SECTION 02 OF 2

2. Design of a Payment System

Key Takeaway: Combine client-to-PSP idempotency keys, single-shard SQL transactions with a transactional outbox, append-only double-entry bookkeeping, and nightly PSP report reconciliation to ensure every cent is accounted for even when networks and services fail.
SECTION 02 DIAGRAMS · SLIDE 1 OF 4Design of a Payment System
ARCHITECTURE / DATA FLOW DIAGRAM
score 0.02Risk checkfraud scoretoken onlyCheckoutsucceededPayment DBSQLPayment serviceLedgerdouble entryPayment executorWalletmerchant balancesPSPexternal

High-Level Architecture, API, and Relational Schema

When a buyer submits checkout, the client first exchanges raw card details directly with the PSP for a card_token, then calls POST /payments carrying an Idempotency-Key UUID generated once per checkout intent. The Payment Service immediately persists the payment record with status = 'PENDING' in Payment DB before executing any external call so a subsequent crash never loses track of the order. A synchronous Risk Check service scores device, velocity, and behavioral signals to block stolen cards before incurring PSP fees or chargebacks, after which a decoupled Payment Executor invokes the PSP asynchronously using non-blocking I/O. Once the PSP responds (either inline or minutes later via an HMAC-signed POST /webhooks/psp callback for 3D-Secure/async payment methods), downstream Ledger and Wallet services record the financial movements.

TableCore ColumnsConstraints & Concurrency ControlRole in Architecture
paymentspayment_id (PK), idem_key, order_id, amount_cents, currency, statusUNIQUE(idem_key) stops duplicate checkout submissions.Lifecycle state machine (PENDING -> SUCCEEDED / FAILED).
ledger_entriesentry_id (PK), payment_id, account_id, direction, amount_cents, created_atStrict append-only (INSERT only; zero UPDATE/DELETE).Immutable double-entry audit trail proving debits = credits.
walletsmerchant_id (PK), balance_cents, versionOptimistic concurrency (WHERE version = expected_v).Fast read model of payable merchant balances for dashboards and payouts.
outboxevent_id (PK), payment_id, event_type, payload, sentWritten atomically in the same SQL transaction as payments.Eliminates dual-write loss between SQL commits and message-queue publishing.
POST /payments
Headers: Idempotency-Key: 8f14e45f-ceea-467a-9a11-7d3b2c99c104
Body:    { "order_id": "ord_902", "amount_cents": 5000, "currency": "USD", "card_token": "tok_visa_491a" }

GET  /payments/{payment_id}
->   { "payment_id": "pay_771", "status": "PENDING" | "SUCCEEDED" | "FAILED" }

POST /webhooks/psp
Headers: X-PSP-Signature: t=1763556000,v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd
Body:    { "psp_charge_id": "ch_391", "idem_key": "pay_771", "status": "succeeded", "amount_cents": 5000 }

Two-Hop Idempotency and Double-Entry Bookkeeping

Preventing double charges requires enforcing idempotency across two distinct network hops. On the first hop (Checkout -> Payment Service), if a user double-taps Pay on a flaky mobile connection, both requests carry the identical Idempotency-Key; the first inserts a payments row while the second hits the UNIQUE(idem_key) constraint, waits for or reads the existing record, and returns the exact same result. On the second hop (Payment Executor -> PSP), we pass our internal payment_id as the PSP's idempotency key so that retrying after a 2 s socket timeout causes the PSP to return the original charge rather than billing the card twice. Once succeeded, we record funds using Double-Entry Bookkeeping rather than mutating a single balance integer: every financial event writes two or more immutable ledger rows whose debits equal credits, making any rounding bug or partial write immediately detectable as an arithmetic imbalance.

Ledger EntryAccountEntry TypeAmount (USD)Financial Meaning (`$50.00` Order, `3%` Fee)
Entry 1PSP_Clearing_ReceivableDebit$50.00External PSP now owes the platform $50.00 upon T+1 settlement.
Entry 2Merchant_Payable (Seller)Credit$48.50Platform owes the seller $50.00 - 3% ($48.50) at next payout.
Entry 3Platform_Fee_RevenueCredit$1.50Platform recognizes $1.50 (3%) take-rate revenue.
Invariant CheckSUM(Debits) == SUM(Credits)Balanced$50.00 == $50.00Zero cents created or destroyed; refunds append opposite entries.
Interview Signal: Always store monetary amounts as exact integer minor units (amount_cents BIGINT, e.g., 5000 for $50.00) paired with an ISO-4217 currency code (USD, JPY), or fixed-precision NUMERIC(19,4)—never IEEE-754 floating-point (FLOAT/DOUBLE), where 0.1 + 0.2 != 0.3 breaks ledger balance checks.

Timeout Recovery, Exponential Backoff, and the Transactional Outbox

When a PSP request times out, the executor does not know whether the issuing bank placed the hold or dropped the packet. The cardinal rule is never to mark a timed-out payment FAILED on a guess: its status remains PENDING, and the executor enqueues a retry job carrying the original payment_id idempotency key onto a dedicated Retry Queue with exponential backoff (1 s, 2 s, 4 s, 8 s plus jitter) so worker threads remain free and struggling PSPs are not hammered. If retries continue failing past a threshold while no signed webhook arrives, the job routes to a Dead Letter Queue (DLQ) for automated PSP status-API polling or human investigation. Similarly, updating payments.status = 'SUCCEEDED' and publishing events to Ledger, Wallet, and Notifications avoids the classic Dual-Write Problem via the Transactional Outbox Pattern: the SQL transaction commits both the payments update and an outbox row atomically, and an asynchronous Relay publishes outbox rows to the Event Queue (at-least-once) while downstream consumers deduplicate by payment_id.

  • Pending-Until-Proven Rule: Every payment terminates in SUCCEEDED, FAILED (confirmed decline), or an actively monitored DLQ—nothing is silently dropped.
  • Webhook Convergence: Asynchronous payment methods (3DS challenges, bank transfers) and race conditions resolve cleanly when the HMAC-verified PSP webhook settles the PENDING row directly.
  • Outbox Crash Safety: If the Relay crashes after publishing to Kafka/SQS but before marking outbox.sent = true, it republishes on restart; idempotent Ledger and Wallet consumers ignore already-applied payment_id events.
  • Consumer Isolation: Slow downstream email receipts or analytics consumers lag independently on the Event Queue without adding latency to checkout.

Nightly Settlement Reconciliation and Design Scorecard

Internal database constraints and double-entry invariants prove our own services are self-consistent, but they cannot detect bugs at the boundary with the outside banking world—such as a missed webhook, a fee-calculation discrepancy, or an unrecorded PSP refund. Every night, the PSP delivers a batch Settlement Report enumerating every captured charge, refund, and fee settled for that day. A batch Reconciliation Engine joins our internal ledger_entries against the PSP settlement report line by line (for example: p1 = $50.00 matches; p2 = $20.00 matches; p3 = $75.00 internal vs $57.00 PSP flags an amount mismatch; p4 = $12.00 internal vs missing in PSP flags a phantom pending/succeeded charge). Discrepancies route to a finance exception queue where automated rules or analysts post explicit compensating adjustment entries to close the gap.

Quality RequirementArchitectural MechanismsPrimary Failure Mode Mitigated
ReliableIdempotency-Key, exponential Retry Queue, DLQNetwork retries double-charging cards or PSP timeouts dropping orders.
ConsistentSharded SQL ACID, debits = credits ledger, Outbox relayCrash between payment update and ledger/wallet credit (dual-write drift).
SecureClient-side PSP card tokenization, HMAC-signed webhooks, Risk CheckServer-side PAN data breach, forged webhook callbacks, stolen-card fraud.
AvailableStateless Payment Service, multi-AZ SQL replicas, async queuesSingle-node hardware crash or slow email/PSP calls starving checkout.
AuditableImmutable append-only ledger_entries, nightly reconciliation, 7-yr storeIn-place balance tampering or silent divergence from bank settlement totals.
ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (3)

CHECKPOINT QUIZ #1

In "Introduction to the Payment System" (Functional and Non-Functional Requirements), how does a production-grade design handle "Functional" (Engineering Implication)?

CHECKPOINT QUIZ #2

In "Design of a Payment System" (High-Level Architecture, API, and Relational Schema), how does a production-grade design handle "ledger_entries" (Role in Architecture)?

CHECKPOINT QUIZ #3

Before wrapping up a 45-minute design round on "Design a Payment System", which core engineering objective should your architecture explicitly satisfy?