Cash, Balances, and Double-Entry Ledgering

Learning outcomes

A brokerage holds other people’s money, and the law, the customer, and the auditor all expect that money to be accounted for to the cent, at every moment, forever. This page teaches cash balances and ledgering as a consistency problem, anchored in the exact scenario Trade Republic documents: a service updates a cash balance and must notify downstream via Kafka, and a crash between the two leaves the database up to date and downstream unaware.

After studying this page, you can:

• Explain why a mutable balance column alone cannot represent money safely, and why an append-only ledger is the correct primitive.
• State the double-entry invariant and prove why it catches every class of accounting error.
• Design idempotent deposits and withdrawals using idempotency keys, and explain why at-least-once delivery is unacceptable for money movement.
• Diagnose and fix the dual-write problem between a balance update and a downstream notification, using the transactional outbox.
• Defend ledger correctness through reconciliation against external settlement records.

Before we dive in

A customer deposits one hundred euros, and the app shows their balance go up. That sentence hides the most dangerous simplification in financial engineering. There is no single number called a balance that you increment. There is a ledger of entries, credits and debits, and the balance is the sum of those entries at a moment in time. The reason no serious financial system stores a mutable balance as the source of truth is that a mutable number leaves no trail: you cannot tell why it has its current value, you cannot detect that a credit was applied twice, and you cannot reconstruct it if it is corrupted. The ledger is the truth, and the balance is a cache.

The double-entry discipline grew out of exactly these failures. Merchants in the Renaissance who recorded only a running total kept losing money to errors and fraud they could not trace. Double-entry bookkeeping, recording every transfer as two offsetting entries so the total is conserved, gave them an invariant that made errors visible. A credit without a matching debit, or a set of entries that does not sum to zero, is mathematically wrong, and the error is detectable by inspection. A modern ledger is the same idea in software: an append-only sequence of entries that always conserves the total, with balances computed from the entries, never stored as mutable state.

The engineering problem this concept exists to solve is the gap between the ledger and the rest of the system. The ledger lives in a database. The deposit was initiated by an external event, a bank transfer, that arrived as a message. The customer’s notification and the downstream services that care about the balance are separate again. Connecting these without ever losing or duplicating a cent, across crashes and retries, is the consistency problem at the heart of brokerage engineering, and the Trade Republic outbox work documents it as precisely as any production writeup can.

Breaking it down

1. Why a balance is not enough

Begin with the wrong design, because seeing why it fails teaches what the right design must provide. Suppose you store each customer’s balance as a column in a row, and a deposit does an update that adds to it. This design has three fatal gaps.

The first is no history. A balance of three hundred euros tells you nothing about how it got there. Was it one deposit of three hundred, or three deposits of one hundred, or a deposit of five hundred and a withdrawal of two hundred? Without the entries, you cannot answer audit questions, you cannot reverse a specific erroneous credit, and you cannot prove to a regulator that the money arrived legitimately.

The second is no idempotency. If the deposit message is delivered twice, as at-least-once systems deliver, the balance is incremented twice. The balance column has no way to recognize that this deposit was already applied, because it carries no record of any specific deposit.

The third is no conservation invariant. Nothing in a set of mutable balance rows forces the total money in the system to stay constant. A bug that credits one customer without debiting anywhere, or that debits the source but never credits the destination, creates or destroys money silently. There is no property you can check that would catch it.

2. Double-entry, the invariant that protects money

The double-entry ledger fixes all three gaps with one discipline. Every money movement is a transaction of two or more entries, each entry crediting or debiting a named account, and the entries in every transaction sum to zero. Money is never created or destroyed, only moved between accounts. This single invariant, that every transaction balances, is what makes errors visible.

A deposit of one hundred euros is not one entry. It is two: a credit of one hundred to the customer’s cash account, and a debit of one hundred from an external-cash, or clearing, account that represents the source of the funds. A withdrawal is the reverse. A trade settlement moves money from cash to a securities-settlement account and back. The total across all accounts is invariant; only the distribution changes.

The power of the invariant is that it catches errors by construction. A transaction that does not sum to zero is rejected by the ledger before it is ever written. A duplicate application of the same deposit is prevented by the idempotency key on the transaction, not by inspecting balances. A query that asks where the money is can be answered by summing entries by account, and if the sum across all accounts differs from the known total, the discrepancy points exactly at a corrupted or missing entry. Double-entry is not a bookkeeping convention bolted onto software, it is the correctness model that makes a ledger trustworthy.

3. Idempotent deposits and withdrawals

Money movement in a distributed system means messages, and messages are delivered at-least-once. A deposit notification from the bank will arrive, and on any transient failure it will be redelivered. A withdrawal request from the customer’s device will be retried by the client. An at-least-once world is unacceptable for money, because applying the same credit twice is a direct loss.

The tool is the idempotency key. Every money movement carries a client- or source-supplied key that uniquely identifies that specific operation. The ledger stores the key on the transaction and refuses to apply a second transaction with the same key. A redelivered deposit is recognized as a duplicate and is acknowledged without re-applying the credit. The balance changes exactly once for each distinct operation, regardless of how many times the message is delivered.

The two non-obvious points in this structure are where the duplicate check happens and what seen means. The check happens inside the same database transaction as the append, so two concurrent redeliveries cannot both pass the check. And seen means the key has been committed, not merely received, so a crash after the check but before the commit does not leave a key marked as seen with no transaction applied. The idempotency guarantee is only as strong as the atomicity that backs it.

4. The balance update plus outbox consistency problem

Here is the exact scenario Trade Republic documents, and it is the centerpiece of this concept. A service updates a customer’s cash balance and must notify downstream services via Kafka. The naive implementation performs the database update and the Kafka publish as two separate operations. The hazard is the crash between them: if the service crashes after the database commits but before Kafka accepts the publish, the database is up to date and downstream is unaware. The customer’s balance changed, but every downstream system that should react to it, notifications, risk, analytics, saw nothing.

The deeper reason this is hard is that Kafka does not participate in a PostgreSQL transaction. You cannot wrap the balance update and the Kafka publish in one atomic unit, because the Kafka produce is a network call to a separate system. Trying to hold the database transaction open across the Kafka call creates a long-running transaction that blocks vacuum and degrades the database, and it still does not guarantee atomicity, because Kafka can accept the produce and then the commit can fail. This is the dual-write problem: two independent systems, no shared transaction, and a crash window between them.

The correct fix is the transactional outbox. Instead of publishing to Kafka directly, the service writes a row to an outbox table in the same database transaction as the balance update. Either both the balance change and the outbox row commit, or neither does, so the database is never in a state where the balance moved and no event exists. A separate process, at Trade Republic a Debezium connector streaming the PostgreSQL write-ahead log, reads the outbox rows and publishes them to Kafka with low latency and near-zero extra load on the table. The outbox row is the durable intent to notify; the publisher is the mechanism that delivers it.

The outbox does not eliminate every problem, it relocates them. The outbox row must be published exactly once or at-least-once with idempotent consumers, the ordering of events matters, and a poison-pill message can stall the publish transaction. These are the documented outbox incidents at Trade Republic: sorting by a created_at timestamp across pods broke total order, so the correct key is a database sequence id; a partial index on id where published_at is null is the right index, indexing published_at is useless; picking too many rows per publish iteration opens a long-running transaction that blocks autovacuum, so about one hundred per batch is the sweet spot; and poison pills stall the whole publish transaction unless messages are grouped by key so independent groups retry in isolation. The outbox is the right pattern, and these are the operational realities of running it at brokerage scale.

5. Reconciliation against external settlement

The internal ledger is one notion of truth. The external settlement system, the clearing house and the bank, is another. They must agree, and the mechanism by which a brokerage discovers disagreement is reconciliation. Reconciliation is not optional polish; it is the feedback loop that catches every error the real-time path missed.

A reconciliation job compares the internal ledger against an external record, a clearing-house file, a bank statement, a custodian report, on a schedule, typically end of day. Where the two agree, the ledger is confirmed. Where they differ, the difference is a break, and a break is a hard signal. A break can mean a lost event, a duplicated entry, a mis-mapped identifier, a settlement failure, or a genuine counterparty default. The reconciliation job does not fix the break; it surfaces it so a human or a defined exception process can investigate and correct it before the customer is affected.

The design implication is that the internal ledger must be reconstructable and auditable to the entry. A break investigation may require asking, for a specific account and a specific day, every entry that touched it, in order, with its source. An append-only, double-entry ledger answers this question directly: the entries are the record, and the balance is their sum. A mutable-balance design cannot answer it at all, which is another reason the ledger is the truth and the balance is a cache. Reconciliation is the external pressure that forces the internal design to be correct, because a design that cannot be reconciled cannot survive an audit.

Common mistakes candidates make

• Storing a mutable balance as the source of truth. The balance is a cache derived from the append-only ledger. Treating it as truth loses history, idempotency, and the conservation invariant.
• Publishing to Kafka directly alongside a balance update. This is the dual-write, and a crash after the commit leaves downstream unaware. Use the transactional outbox with a CDC publisher.
• Checking the idempotency key outside the database transaction. Two concurrent redeliveries can both pass the check and both apply. The check and the append must be one transaction.
• Sorting outbox events by a created_at timestamp across pods. Pod clocks are skewed and break total order. Use a database sequence id as the ordering key.
• Forgetting the conservation invariant. A ledger transaction whose entries do not sum to zero creates or destroys money. Enforce the invariant at append time, reject any transaction that violates it.
• Treating reconciliation as optional. Without reconciliation against external settlement, errors in the real-time path go undetected until a customer complains or an audit fails.

Mastery Questions