On this path · Distributed Primitives 3. Consistency, Transactions, and Rollbacks
Consistency, Transactions, and Rollbacks
Two-phase commit versus saga versus outbox, isolation levels, and what handling rollback actually means for money.
Learning outcomes
A documented reason candidates fail the Trade Republic loop is “not handling transaction rollbacks.” That phrase sounds simple, but it names a deep failure: the candidate designs a happy path where every operation succeeds, and has no answer for what the system’s state looks like when a transaction aborts halfway through a multi-step operation. This concept is about the judgement a senior engineer applies to consistency, the isolation guarantees a database actually gives you, and what it means to handle a rollback when money and external systems are involved.
After studying this page, you can:
- State what ACID guarantees and what it does not, especially the isolation gap between levels.
- Distinguish read committed, repeatable read, and serializable isolation, and name the anomaly each allows (lost update, write skew).
- Explain why “not handling transaction rollbacks” is a documented failure mode and what handling a rollback actually requires.
- Compare two-phase commit, saga, and the outbox pattern as distributed consistency strategies, and defend why the outbox wins for a brokerage.
- Describe what rollback means for money: compensating actions, idempotent retry, and the absence of partial external effects.
- Tie consistency theory back to the dual-write problem and the outbox invariant.
Before we dive in
Imagine a withdrawal. A customer asks to move 500 euros out of their Trade Republic account to their external bank. The operation has several steps: check the balance is sufficient, debit the internal cash ledger, instruct the external payment rail to send the funds, and record the transaction for the statement. The engineer’s first instinct is to wrap the ledger debit and the payment instruction in one database transaction, so they commit or roll back together. Then the engineer learns that the external payment rail is a separate service reached over a network, and a database transaction cannot roll back a network call that has already been sent.
This is the moment the happy-path design breaks. If the database transaction commits the debit but the payment instruction already left and later fails, the customer’s balance is down 500 euros and no payment is on its way. If the payment instruction is sent before the commit and the commit then rolls back, the payment rail sends money the ledger never debited. The engineer who has only thought about the success case has no answer for either failure, because the tools that made the happy path clean, transactions and commits, do not extend across the system boundary. The documented interview failure, “not handling transaction rollbacks,” is exactly this: a design whose rollback behavior is undefined for any step that is not a local database write.
The deeper subject this forces is distributed consistency. When an operation spans a database and an external system, no single transaction covers the whole thing, so you must choose a strategy for keeping the parts consistent despite independent failures. The strategies have names (two-phase commit, saga, outbox), and the choice between them is a judgement about latency, availability, and the cost of inconsistency. This concept teaches that judgement from the brokerage scenario that produces it.
What a transaction actually guarantees
A database transaction gives you atomicity, consistency, isolation, and durability. Atomicity means the transaction’s changes all happen or none happen: a commit makes them permanent, a rollback discards them. Consistency means the transaction moves the database from one valid state to another, respecting constraints. Durability means a committed transaction survives a crash. Isolation is the subtle one, and it is where most production bugs live, because isolation is not a single guarantee but a spectrum.
Atomicity and durability are binary: a transaction commits or it does not. Isolation is a choice of level, and each level permits a specific set of anomalies where concurrent transactions interfere with each other. The level you pick determines which races your code must defend against.
The mistake is to assume that wrapping operations in BEGIN and COMMIT makes them safe from concurrency. It does not, unless you pick the right isolation level and write your queries to match. The database’s default isolation is often weaker than engineers assume, and the anomalies that weaker levels allow are exactly the bugs that corrupt money and holdings.
Isolation levels and the anomalies they allow
PostgreSQL offers three isolation levels that matter in practice: read committed (the default), repeatable read, and serializable. Each permits a different set of anomalies, and the cost of stronger isolation is more overhead and more serialization failures the application must retry.
Read committed is the default. It guarantees that a statement sees only data committed before the statement began, so you never read uncommitted (dirty) data. It does not guarantee that two statements in the same transaction see the same snapshot. A transaction that reads a balance, does some work, and reads the balance again can see a different value the second time if another transaction committed in between. This non-repeatable read is legal under read committed, and it is the source of many race conditions.
Repeatable read gives the transaction a single snapshot taken at its first read, so every statement in the transaction sees the same view of the database. Non-repeatable reads disappear. This is the level most financial code wants, because a transaction that computes a balance and then acts on it needs the balance to be stable for the transaction’s duration. The cost is that the transaction may fail to commit if it conflicts with a concurrent transaction that committed first, which the application must handle by retrying.
Serializable is the strongest level. It guarantees that the outcome of concurrent transactions is equivalent to some serial execution, meaning the system behaves as if transactions ran one at a time in some order. PostgreSQL implements serializable through predicate locking and conflict tracking, which detects when two concurrent transactions could not both have run in any serial order, and aborts one with a serialization failure. The application must be written to retry on that failure. Serializable is the safest but the most expensive, and at high contention it causes many retries.
The lost update and write skew anomalies
Two anomalies are worth understanding in depth because they are the ones that corrupt financial data, and interviewers probe them directly.
A lost update happens when two transactions read a value, compute a new value based on it, and write it back, and one write clobbers the other. The classic example is two concurrent withdrawals from the same account. Transaction A reads a balance of 1000, computes 1000 minus 500, and writes 500. Transaction B reads the same 1000 before A commits, computes 1000 minus 300, and writes 700. The final balance is 700: A’s withdrawal of 500 is lost. Under read committed, this anomaly is allowed. Under repeatable read in PostgreSQL, the database detects the conflict and aborts the second writer, which is why repeatable read is the practical choice for balance updates.
The fix is to make the write conditional on the value not having changed, which is optimistic locking, or to lock the row for the duration of the transaction, which is pessimistic locking. These are covered in detail in the concurrency concept, but the principle is that the database must be told that the read and the write are a unit, so it cannot let another transaction slip between them.
Write skew is subtler and is the anomaly that repeatable read does not catch in some databases (though PostgreSQL’s repeatable read does catch lost updates, it does not catch all write skew; that requires serializable). Write skew occurs when two transactions read overlapping data, each writes based on what it read, and the combined result violates a constraint that neither transaction alone violated. The classic example is an on-call constraint: two doctors are on call, the rule is at least one must remain, and two transactions each let one doctor go off duty after seeing that the other is still on call. Each transaction alone is valid, but together they violate the invariant. Repeatable read does not always detect this because the transactions write different rows; serializable does, because it tracks the read-write dependency.
Two transactions that write different rows can both commit under repeatable read even when their combined effect violates a constraint, because each transaction’s write does not conflict at the row level. Only serializable isolation, which tracks predicate-level read dependencies, catches it. If your invariant spans multiple rows, you need either serializable or an explicit constraint or lock.
For a brokerage, the money analog is a multi-account constraint: a customer’s total margin across accounts must stay above a threshold, and two concurrent withdrawals from different accounts each pass individually but together breach the threshold. The defense is serializable isolation, or an explicit aggregate check guarded by a lock that covers all the relevant accounts.
Not handling transaction rollbacks
The documented Trade Republic interview failure mode is “not handling transaction rollbacks on failure.” The phrase names a specific, common mistake: the candidate designs a sequence of operations, assumes each succeeds, and has no defined behavior for what happens when one fails after others have already committed or after an external effect has fired.
Consider the withdrawal again. The ledger debit is a local database write, so it can roll back cleanly. The external payment instruction is a network call to a separate system, so it cannot. A candidate who puts the payment instruction inside the database transaction has a rollback problem in both directions. If the transaction commits and the payment later fails, the ledger is debited but no money moved: the customer is missing funds. If the payment instruction fires before the commit and the commit then rolls back, the payment rail sends money the ledger never recorded: the customer is credited externally for a debit that never happened.
Handling the rollback means accepting that the external world is not transactional and designing the operation so that every partial state is recoverable. The strategies for this are the distributed consistency patterns, and the one a brokerage reaches for is the outbox: write the debit and a “send payment” event in one transaction, and let a separate process drive the external rail and retry until it confirms. The debit and the instruction commit together, the instruction is delivered at-least-once, and the external call’s failure never corrupts the ledger because the ledger state is the source of truth and the external call is a driven consequence.
Two-phase commit, saga, and outbox compared
There are three classic strategies for keeping a multi-system operation consistent, and a senior engineer should know all three and why a brokerage picks the outbox.
Two-phase commit, or 2PC, is a protocol where a coordinator asks every participant to prepare, and once every participant confirms it can commit, the coordinator tells them all to commit. It gives true atomicity across systems. The cost is that it is blocking: if the coordinator fails after prepare, the participants hold their locks indefinitely, waiting for a decision they cannot make themselves. 2PC also requires every participant to support the protocol, which most external systems do not. For a brokerage calling an external payment rail, 2PC is simply not available, because the rail does not implement it. 2PC is the right answer for atomicity within a controlled set of internal databases, and the wrong answer for operations that cross system boundaries to third parties.
A saga breaks the operation into a sequence of local transactions, each with a compensating action that undoes it if a later step fails. For the withdrawal, the saga is: debit the ledger, then send the payment; if the payment fails, run the compensation, which credits the ledger back. The saga gives up global atomicity in exchange for availability: there is no distributed lock, so the system never blocks, but the intermediate states (debit done, payment pending) are visible, and the compensation must be correct and idempotent. Sagas are powerful for long-running workflows, and their risk is that the compensation logic is hand-written application code that must perfectly reverse every effect, which is hard to get right and easy to leave buggy.
The outbox, as covered in its own concept, writes the business change and an event in one local transaction, then delivers the event asynchronously. It is the strategy that fits the brokerage case because the external system is a consequence of the local truth, not a co-equal participant. The ledger is authoritative; the payment rail is driven from the ledger. The outbox makes that relationship explicit and durable. Compared to a saga, the outbox has a simpler compensation story: the local transaction is atomic, and the only thing that can partially fail is delivery, which is retried rather than compensated.
A brokerage’s source of truth is its own database: the ledger, the holdings, the orders. External systems (payment rails, exchanges, downstream services) are driven consequences of that truth. The outbox honors that asymmetry by making the local write atomic and treating external effects as delivered events, which avoids both the blocking of 2PC and the hand-written compensations of a saga.
A worked multi-step money flow
To make the strategies concrete, trace a single withdrawal through each. The operation has four logical steps: check the balance is sufficient, debit the internal ledger, instruct the external payment rail, and record the statement entry. Steps two and four are local database writes and can share a transaction. Step three is a network call to a separate system.
Under two-phase commit, all four steps would prepare, then all commit. This fails immediately because the payment rail does not implement the prepare phase, so 2PC is not available for this operation at all. Under a saga, the steps run as a sequence: the ledger debit commits, then the payment instruction is sent, then the statement entry is written. If the payment instruction fails, the saga runs compensations in reverse: credit the ledger back and reverse the statement entry. Each compensation must be idempotent, because a retried compensation must not double-reverse. The intermediate state (debit done, payment not yet confirmed) is visible to any concurrent read, so a balance query during the window sees the debit before the payment confirms.
Under the outbox, the ledger debit and a payment instruction event commit in one transaction, atomically. A separate publisher drives the payment rail from the event, retrying until it confirms. The statement entry is written in the same transaction as the debit. There is no intermediate state where the ledger is debited but no instruction exists: either both committed, or neither did. The only thing that can fail partially is the delivery of the instruction, which is retried, not compensated. The external effect is a consequence of the local truth, delivered at least once.
The key difference from the saga is the visibility of intermediate state. The saga exposes a window where the debit is committed but the external effect is unresolved, and a concurrent reader sees that window. The outbox collapses that window: the local truth and the instruction to act appear together, atomically. For money, where a customer seeing a debited balance for a payment that has not sent is a support ticket, collapsing that window is worth the asynchronous delivery.
Isolation in practice: which level for which operation
The isolation choice is not global; it is per operation, matched to the invariant it protects. A balance update is a read-modify-write on one row, and its anomaly is the lost update. Repeatable read in PostgreSQL catches lost updates by aborting the second writer, so repeatable read with retry is the practical choice for balance and holdings updates. The cost is occasional serialization failures on contention, which the application retries.
A multi-row invariant, like the aggregate margin across accounts, is vulnerable to write skew, which repeatable read does not fully catch. For these, serializable isolation is the correct choice, because it tracks the read-write dependencies across rows and aborts a transaction whose outcome could not occur in any serial execution. Serializable is more expensive and produces more aborts under contention, so it is reserved for the invariants that genuinely span rows, not applied to every transaction as a default.
Use repeatable read with retry for single-row read-modify-writes like balance updates, because it catches lost updates at low cost. Reserve serializable for multi-row invariants like aggregate margins that are vulnerable to write skew, because only serializable tracks cross-row read dependencies. Applying serializable everywhere pays its overhead on transactions that do not need it, and applying read committed to a write-skew-prone operation silently corrupts the invariant.
What rollback means for money
For a database transaction, rollback is mechanical: the engine discards the changes and the state is as if the transaction never ran. For money, rollback is a business concept, and it is rarely a true reversal. Once a payment has left the building, you cannot roll it back; you send a reversing payment. Once a trade has settled, you cannot un-settle it; you book an offsetting trade. Handling rollback for money means designing the operation so that every partial state has a defined, idempotent recovery path, and so that no partial external effect is ever produced.
The principles are three. First, the local ledger is the source of truth, and it is transactional: the debit and the instruction record commit together, or not at all. Second, external effects are driven from the ledger through at-least-once delivery, so a crash retries rather than corrupts. Third, every operation is idempotent: a retried instruction must recognize that it already executed and not double-apply, identified by a stable idempotency key. A payment instruction carried by a stable identifier can be retried any number of times safely, because the rail recognizes the identifier and returns the original result rather than executing twice.
The dual-write problem from the outbox concept is the root of all of this. The moment an operation spans the database and an external system, the database’s rollback no longer covers the whole operation, so the engineer must design the rollback semantics explicitly. The candidate who “does not handle transaction rollbacks” is the one who never noticed the transaction boundary stopped at the database, and who has no answer for the state left behind when an external step fails.
Common mistakes candidates make
The first and most documented mistake is not handling the rollback at all. The candidate designs a clean sequence of writes, wraps them in a transaction, and when asked what happens if an external call fails after the commit, has no answer. The fix is to recognize that the transaction boundary is the database, and that anything outside it needs an explicit recovery design.
The second mistake is assuming the default isolation level is safe. Engineers wrap balance updates in a transaction and believe the transaction prevents races, when read committed (the default) allows lost updates. The fix is to know which anomaly each isolation level permits and to pick repeatable read or serializable for financial correctness, with retry logic for the conflicts the stronger levels surface.
The third mistake is reaching for two-phase commit to solve every distributed consistency problem. 2PC is blocking and requires every participant to support it, which external systems do not. A candidate who proposes 2PC for a payment rail has not checked whether the rail can participate, and has not weighed the blocking cost.
The fourth mistake is writing saga compensations that are not idempotent. A compensation that credits the ledger back must recognize a retried compensation and not double-credit. Without an idempotency key, a retried compensation corrupts the balance. Every compensation must be safe to run more than once.
The fifth mistake is treating a rollback as a true reversal for money that has already left the system. A payment that has been sent cannot be rolled back; it must be reversed by a new, explicit operation. The candidate who says “I will roll back the payment” has not understood that external effects are not transactional.
Defending the design
When an interviewer asks how you keep a multi-system money operation consistent, the defense is layered. Start with the local truth: the ledger debit and the event instruction commit in one database transaction, so the internal state is always atomic. Move to delivery: the instruction is delivered at-least-once through the outbox, so a crash retries rather than loses the event. Move to idempotency: every external operation carries a stable idempotency key, so a retried delivery does not double-apply. Move to recovery: a confirmed external failure triggers an explicit, idempotent compensating action, not a magical rollback.
The follow-up is almost always about isolation. The defense is: balance and holdings updates run under repeatable read at minimum, so the database detects and aborts lost updates, with the application retrying on conflict. For invariants that span multiple rows, serializable isolation or an explicit lock covers the aggregate. The point is that isolation is a deliberate choice, not an accident of BEGIN, and the application is written to retry the conflicts that stronger isolation surfaces.
The final probe is about why not a saga or 2PC. The answer is that 2PC is blocking and unavailable across third-party systems, and a saga’s hand-written compensations are a bug surface that grows with the workflow’s complexity. The outbox makes the local write atomic and treats the external effect as a driven consequence, which fits the brokerage’s asymmetry: the database is authoritative, and the rest of the world follows.
A sharper probe asks what happens when the compensation itself fails. If a payment fails and the reversing credit also fails, the system is stuck in a state the saga model assumes away. The defense is that compensations are themselves retried at-least-once with an idempotency key, and that a compensation that cannot complete is escalated to a reconciliation process, not silently dropped. Reconciliation is the safety net under every eventual-consistency strategy: a periodic job that compares the ledger against the external system, detects divergence, and raises it for manual or automated resolution. No eventually-consistent money system is complete without reconciliation, because retries and compensations can both fail, and the only way to catch the residual divergence is to compare the two systems on a schedule. A candidate who designs compensation without reconciliation has not handled the case where compensation fails, which is the rollback failure mode one layer deeper.
Mastery Questions
Question
What does it mean to 'not handle transaction rollbacks', and why is it a documented Trade Republic interview failure?
Answer
It means designing an operation as a sequence of steps with no defined behavior for the state left behind when a step fails after others have committed or after an external effect has fired. The transaction boundary is the database, so anything outside it (a network call to a payment rail) is not rolled back by the database transaction. A candidate who has only thought about the happy path has no answer for the inconsistent state a mid-operation failure creates.
Sources & evidence7 claims · 3 cited
Rollback failure mode and dual-write backed by interview and Debezium sources; isolation levels, anomalies, saga/outbox comparison, and compensation-for-money reasoning is internal-reasoning.
- Not handling transaction rollbacks on failure is a documented Trade Republic candidate failure mode: designing an operation with no defined behavior for the state left behind when a step fails after others have committed or after an external effect has fired.verified
- Under read committed isolation the database allows lost updates: two transactions can both read a value, compute, and write back, and one update is lost. Repeatable read catches lost updates in PostgreSQL; serializable makes the outcome equivalent to a serial execution but surfaces serialization failures the application must retry.internal reasoning
- Write skew occurs when two transactions read overlapping data, each writes a different row based on what it read, and the combined result violates a constraint neither alone violated; it evades row locks because the writes touch different rows, so only serializable isolation or an explicit aggregate constraint catches it.internal reasoning
- Two-phase commit gives atomicity across systems but is blocking and requires all participants to support the protocol; a saga breaks an operation into local transactions with compensations; the outbox writes the business change and an event in one local transaction then delivers the event at least once. The outbox fits a brokerage because the database is authoritative and external systems are driven consequences.verified
- Handling rollback for money means accepting the external world is not transactional: the local ledger debit and event instruction commit together, external effects are driven at least once through the outbox, and every operation carries a stable idempotency key so retried delivery or compensation does not double-apply.internal reasoning
- The dual-write problem is the root case: an operation spans the database and an external system, so the database rollback no longer covers the whole operation, forcing explicit recovery design for every step outside the transaction boundary.verified
- A money operation that has produced an external effect cannot be truly rolled back; a sent payment must be reversed by a new explicit compensating operation, and a settled trade offset by another trade, so recovery is by idempotent compensation rather than mechanical reversal.internal reasoning
Cited sources
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)
- Streaming Outbox Events from Postgres to Kafka with Debezium · Trade Republic Engineering (Andrei Sukharev)
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)