On this path · Financial Systems 5. Idempotency, Exactly-Once, and Money Safety
Idempotency, Exactly-Once, and Money Safety
Why financial correctness is not at-least-once: dedup, sequence numbers, and poison pills in money movement.
Learning outcomes
A banking app must never charge you twice for one payment, never lose one, and never apply one out of order, even though the networks underneath deliver messages late, duplicated, and reordered. This is the gap between at-least-once delivery and financial correctness, and it is the topic this concept exists to teach. The material anchors in the documented Trade Republic outbox incidents, sequence ordering, and poison pills.
After studying this page, you can:
- Explain precisely why at-least-once delivery is unacceptable for money movement and what exactly-once semantics demand of an application.
- Design idempotency-key deduplication that survives concurrent redelivery and crashes.
- Use monotonic sequence numbers to guarantee event ordering and explain why timestamps fail.
- Diagnose and isolate poison-pill messages so one bad event cannot stall a stream of good ones.
- Combine deduplication, ordering, replay protection, and reconciliation into a defensible money-safety model.
Before we dive in
You tap to buy a stock, the app shows a spinner, the network is flaky, so you tap again. The question that a money-safe system must answer perfectly is: did you buy once or twice? In an at-least-once world, where the network and the message bus redeliver freely, the default answer is twice, and twice is a direct financial loss to you or to the broker. Every financial system is, at its core, a campaign to make the answer come out to exactly once, no matter how many times the underlying messages are delivered.
The reason this is a distinct discipline, and not just a feature of the message bus, is that exactly-once is not a transport property you can buy. The message bus can offer transactions and idempotent producers, and they help, but the final responsibility for applying an operation exactly once belongs to the application that moves the money. The bus does not know that message number seven and message number nine are the same customer intent, because that is a business fact encoded in an idempotency key only the application understands. Exactly-once in finance means exactly-once effect, and the effect is defined at the point where money moves, not where the message travels.
The documented Trade Republic outbox work is the clearest production case study of this discipline. The team operated an outbox that carried balance-change events from the database to Kafka, and they hit a sequence of real incidents: ordering broke when they sorted by timestamps across pods with skewed clocks; long-running publish transactions blocked autovacuum and degraded the database; poison-pill messages stalled the publish and blocked every event behind them; partial indexes bloated until query times exploded; and integer sequences threatened to run out. Each incident is a lesson in how at-least-once assumptions break money safety, and each fix is a piece of the exactly-once model this concept teaches.
Breaking it down
1. Why financial correctness is not at-least-once
Begin with the three delivery guarantees and why only one is acceptable for money. At-most-once means a message may be delivered zero or one times; the failure mode is a lost message, which for money means a deposit that never credits or a trade that never settles. At-least-once means a message may be delivered one or more times; the failure mode is a duplicate, which for money means a credit applied twice or a payment taken twice. Exactly-once means the effect of the message is applied precisely once, regardless of how many times it is delivered.
For most systems, at-least-once with idempotent handling is good enough, because the idempotent handling collapses duplicates into one effect. For money, the requirement is stricter: the effect must be exactly-once, and the proof of it must be reconstructable for audit. The reason at-least-once is unacceptable is not theoretical. Networks partition, consumers crash mid-processing, and producers retry on timeout. Every one of these events, in an at-least-once system, can produce a redelivery, and a redelivery that is not deduplicated is a double-charge. The financial cost is immediate and the regulatory cost, a mis-stated balance, follows.
The diagram makes the central claim of this concept concrete: the client, not the network, owns the idempotency key, and the key is generated once at the moment of intent and reused on every retry. The dedup table is the single place where duplicate intents are collapsed, and because that check shares a transaction with the trade application, two concurrent retries cannot both pass it. Whatever the network does, redeliver, reorder, or stall, the effect on the ledger is one and only one trade.
The shift in mindset that this concept teaches is that exactly-once is not a guarantee you request from your infrastructure. It is a property you build at the application boundary, by making every money-moving operation idempotent and by ordering and replay-protecting the events that drive them. The infrastructure gives you the raw materials, transactions, sequences, dedup tables, and you assemble them into an exactly-once-effect guarantee for the specific operations that move money.
2. Idempotency keys, the dedup primitive
The first raw material is the idempotency key. Every money-moving operation carries a key that uniquely identifies that specific intent, and the system refuses to apply a second operation with a key it has already applied. A redelivery arrives with the same key, is recognized as a duplicate, and is acknowledged without re-applying the effect. The customer who tapped twice generated two client requests, but the system applied the underlying trade or transfer exactly once because both requests carried the same client-supplied idempotency key.
The strength of the guarantee depends entirely on where the dedup check happens. The check must be inside the same database transaction as the effect it guards, so that two concurrent redeliveries cannot both pass the check before either commits. If the check is a separate query before the transaction, two redeliveries can race through the gap, both see the key as unseen, and both apply. The dedup state and the effect must commit atomically, or the dedup is advisory rather than guaranteed.
The subtle point is what the key identifies. It identifies the customer’s intent, the specific buy or transfer they meant to perform, not the message that carries it. Two taps that express the same intent carry the same key and are deduplicated. Two genuinely separate buys carry different keys and are both applied. Getting this right means the client must generate the key at the moment of intent and reuse it on retry, which is a client-side discipline as much as a server-side one.
The Trade Republic partitioning benchmark underscores the operational scale of dedup. A dedup table that records every applied idempotency key grows without bound and becomes a bottleneck. Their finding, that a UUID-keyed dedup table partitioned into sixteen partitions performed better under hash partitioning than list partitioning, and that hash partitioning preserved primary keys needed for logical replication, is the kind of detail that separates a design that works in a demo from one that works in production. Dedup is not a single row in a table; at brokerage scale it is a sharded, vacuumed, performance-tuned subsystem in its own right.
3. Monotonic sequence numbers and ordering
Deduplication handles duplicate application. Ordering handles the sequence in which distinct operations are applied, and for money, ordering is not a cosmetic concern. A withdrawal applied before the deposit that funds it is an overdraft; a trade settled before an earlier trade that established the position is a short sale. The events that drive money movement must be applied in the order they were intended, and that order is defined by a monotonic sequence number, not by arrival time.
The documented Trade Republic outbox incident is the textbook case. The team initially sorted outbox messages by their created_at timestamp to determine publish order, and this broke total ordering because the application ran across multiple pods whose clocks were skewed relative to each other. A message created later on a pod with a slow clock could sort before a message created earlier on a pod with a fast clock, so the published order did not match the real order of events. The fix was to order by a database sequence id, a single monotonically increasing counter generated by the database, which is correct regardless of how many pods are involved because the database is the one source of the sequence.
This lesson generalizes to every place a distributed system needs total order: when the participants do not share a clock, do not sort by wall time. Use a sequence from a single source, whether that is a database identity column, a single-threaded sequencer, or a consensus-given log position. The matching engine uses the same principle for ordering equal-priced orders, and the outbox uses it for ordering published events, because both face the same fundamental problem: clocks lie, sequences do not.
4. Poison-pill messages and replay protection
A poison-pill message is one that cannot be processed successfully, no matter how many times it is retried. The cause is usually a malformed payload, an incompatible schema version, or a reference to a missing entity. The danger is not the message itself, which is broken, but what it does to every message behind it. If processing is strictly ordered, and the poison pill sits at the head of the queue, retrying it forever blocks every subsequent message, and the entire stream stalls.
The documented Trade Republic outbox incident describes exactly this: a poison-pill message stalled the whole publish transaction, because the publish ran as one atomic batch and a single failure rolled back or blocked the batch. The fix was to group messages by key so that independent groups could be retried in isolation. A poison pill in one group blocks only that group, while groups with different keys continue to publish. This trades the simplicity of one transactional batch for operational resilience, and it is the right tradeoff, because a stalled outbox is a production emergency while a single stuck key is a bounded exception.
Replay protection is the companion concern. When a consumer restarts and begins replaying events from an earlier offset, it must not re-apply events that were already applied. Replay protection is deduplication across time, not just across concurrent redelivery, and it uses the same idempotency-key mechanism: every event that has an effect carries a key, and reprocessing an already-applied key is a no-op. A consumer that can safely replay from any offset is a consumer that can be restarted, upgraded, and recovered without fear, which is the operational property that lets a money-moving system be maintained at all.
The diagram captures the tradeoff that makes key-grouped retry worth its complexity. Without grouping, the poison pill in group C would sit at the head of one transactional batch and block groups A and B indefinitely, stalling the whole stream for a single bad message. With grouping, A and B continue to publish while only C is stuck, and C is routed to a dead-letter path for investigation. The cost is a more elaborate publisher; the benefit is that one malformed event degrades a single key rather than the entire money-moving pipeline.
5. Reconciliation against external settlement
Idempotency, ordering, and replay protection make the internal system correct with high probability, but high probability is not enough for money. The final layer of money safety is reconciliation against an external authority, the clearing house, the bank, the custodian, whose records are independently produced and independently audited. Reconciliation is the closed-loop check that catches the error the real-time path missed.
The discipline is simple to state and demanding to run. On a schedule, the internal ledger is compared against the external record, and every difference is a break. A break is not a failure of the system, it is the system working as designed: it is the signal that surfaces a lost event, a duplicated entry, a settlement that did not happen, or a counterparty default, so that a human or a defined exception process can investigate and correct it before the customer is affected. A money-moving system without reconciliation is a system that discovers its errors from customer complaints, which is the worst possible feedback loop.
The design implication, as the ledgering concept established, is that the internal ledger must be reconstructable to the entry. Reconciliation is only possible if you can answer, for any account and any period, the exact sequence of entries that produced its balance. An append-only, double-entry ledger answers this directly. Reconciliation and the ledger design are two sides of one coin: the external pressure of reconciliation forces the internal design to be auditable, and the auditable design makes reconciliation tractable.
6. What exactly-once really means
Bring the pieces together into the precise definition an interviewer will probe. Exactly-once, in the only sense that matters for money, means exactly-once effect: an operation produces its effect on the system exactly once, regardless of how many times the underlying message is delivered. It does not mean a message is delivered exactly once, which is a transport property and is generally impossible to guarantee, and it does not mean an operation is processed exactly once, which is a performance property. It means the money moves once.
Exactly-once effect is built from four raw materials. Idempotency keys collapse duplicates into one effect. Monotonic sequence numbers apply distinct operations in the intended order. Replay protection makes reprocessing from any offset safe, so recovery and upgrade do not double-apply. Reconciliation against external authority catches the residual errors the real-time path missed. Each raw material handles a different failure mode, and together they raise the probability of exactly-once effect high enough that the residual risk is acceptable for money, with reconciliation as the backstop that bounds the residual to what an audit can catch.
A senior follow-up is to ask you to justify the cost. Exactly-once effect is not free: the dedup table grows, the sequence is a serialization point, the outbox adds latency, and reconciliation is a daily operational burden. The strong answer acknowledges the costs and names the alternative, which is accepting at-least-once and paying for the resulting double-charges and lost events, which are more expensive and, for a regulated broker, legally unacceptable. The cost of exactly-once is the cost of being trusted with money, and a senior engineer frames it as a deliberate tradeoff, not as a free guarantee.
Common mistakes candidates make
- Assuming the message bus provides exactly-once. It does not, not as an effect guarantee. Exactly-once effect is built at the application boundary with idempotency keys and ordering. The bus provides raw materials, not the finished guarantee.
- Checking the idempotency key outside the transaction. Concurrent redeliveries race through the gap and both apply. The check and the effect must be one transaction.
- Ordering events by wall-clock timestamp across pods. Pod clocks are skewed and break total order. Use a database sequence id or a single-source sequencer.
- Running the outbox publish as one giant transaction. A poison pill stalls the whole batch and blocks every event. Group by key so independent groups retry in isolation.
- Treating reconciliation as optional or as a reporting feature. Reconciliation is the closed-loop check that catches real-time errors. Without it, errors surface as customer complaints.
- Believing exactly-once is free. It has a real cost in storage, latency, and operational burden. The alternative, at-least-once for money, is more expensive and often illegal.
Mastery Questions
Question
Why is at-least-once delivery unacceptable for money movement, and what does exactly-once mean in the sense that matters?
Answer
At-least-once permits redelivery, and a redelivered money operation that is not deduplicated is a double-charge or a lost reversal. Exactly-once in finance means exactly-once effect: the operation produces its effect precisely once regardless of delivery count. It is not a transport guarantee from the bus; it is built at the application boundary with idempotency keys, sequence ordering, replay protection, and reconciliation.
Sources & evidence4 claims · 2 cited
Exactly-once effect definition, idempotency keys, monotonic sequence ordering, poison pills, replay protection, reconciliation. Grounded in src_tr_outbox1 (sequence ordering, poison pills), src_tr_partitioning (UUID dedup partitioning benchmark). Exactly-once decomposition framing marked internal-reasoning.
- Exactly-once in finance means exactly-once effect, not a transport guarantee; it is built at the application boundary from idempotency keys, monotonic sequence numbers, replay protection, and reconciliation.verified
- Sorting outbox messages by created_at across pods broke total ordering because pod clocks are skewed; the database sequence id is the correct ordering key, which is the lesson that clocks lie and sequences do not.verified
- A poison-pill message that cannot be processed stalls the whole publish transaction and blocks every event behind it; the fix is to group messages by key so independent groups retry in isolation.verified
- A UUID dedup table partitioned into sixteen partitions performed better under hash partitioning than list partitioning and preserved primary keys needed for logical replication, illustrating that the dedup table is itself a sharded, tuned subsystem at brokerage scale.verified
Cited sources
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)
- Postgres partitioning performance: Hash vs. List · Trade Republic Engineering (Sadeq Dousti)