On this path · Coding Patterns 4. Concurrency Bug Hunting and Testing
Concurrency Bug Hunting and Testing
The portfolio-value race condition debug, hypothesis-driven debugging, and property testing.
Learning outcomes
Trade Republic’s documented debugging prompt is the kind that surfaces only under load: a user’s portfolio value is occasionally wrong after a trade, the code sums executed trades, and you must analyze the snippet for race conditions and logic errors, then describe how to test the fix. This concept teaches the hypothesis-driven method a senior engineer uses to find such a bug, the catalog of concurrency flaws to check against, the fixes that actually close each flaw, and the testing strategy that proves the fix holds rather than merely passing once. The portfolio-value prompt is the vehicle, but the method generalizes to any occasionally-wrong concurrent computation.
After studying this page, you can:
- Apply a hypothesis-driven debugging loop to an occasionally-wrong concurrent result.
- Identify the lost-update, non-atomic read-modify-write, check-then-act, and visibility flaws in a snippet.
- Choose between atomic compare-and-swap, locks, immutable snapshots, and idempotent appends and defend each.
- Design a test that reproduces a race, including property-based and stress tests with many threads.
- Use a race detector and explain what it catches and what it misses.
- Explain why a passing test never proves a concurrency fix, only raises confidence.
The buggy snippet and the symptom
The buggy code holds the portfolio value as a mutable shared field, updates it on every executed trade, and reads it on every request. The symptom is that the value is occasionally wrong by exactly one trade’s amount, which is the tell that a single update is being lost.
The snippet has at least three distinct flaws, and the senior skill is to name all three rather than jump to the first one spotted. The first is the lost update from a non-atomic read-modify-write. The second is a check-then-act window if the real code (the prompt mentions summing trades, which often involves a conditional) reads a value, decides something, and acts. The third is a memory-visibility problem, because on the JVM a plain var gives no guarantee that one thread’s write is ever seen by another without a happens-before relationship. Each flaw demands a different fix, and a fix that closes one but not the others leaves the bug in place.
Forming hypotheses about the race
The method begins not with a fix but with a hypothesis. The symptom is that the value is occasionally off by one trade’s amount. That specific signature points at a single lost update, not at a logic error or a bad formula. A logic error would be consistently wrong; a lost update is occasionally wrong, and the error magnitude matches a trade amount.
From that signature, the candidate hypotheses are, in order of likelihood for this symptom: a lost update in the read-modify-write on value, a visibility problem where a write is not seen by a later read, and a check-then-act window if the real code has a conditional. Each hypothesis predicts a different failure pattern. A lost update predicts the value is too small (one addition lost) or occasionally too large (a torn read on a non-atomic long). A visibility problem predicts the value is stale, sometimes by many trades. A check-then-act predicts the value reflects a decision based on stale data.
The next step is to design a test that distinguishes the hypotheses. The cleanest test is to run many threads adding known amounts in parallel and to assert that the final value equals the sum of all amounts. A lost update makes the final value smaller than the expected sum. A visibility problem makes reads during the run observe stale values. The test does not need to match production traffic exactly, it needs to maximize the chance of the suspected interleaving, which means high parallelism and enough operations that the race fires many times.
This is the core reason concurrency bugs are intermittent: each individual interleaving is unlikely, but over millions of operations the unlikely becomes certain. The test that reproduces the bug is the one that runs enough operations across enough threads to make the unlikely fire within the test window.
Identifying the lost-update and check-then-act flaws
The lost update is the clearest flaw. The line value = value + amount is three operations: read value, add amount, write value. The JVM executes them as separate steps, and between any two steps another thread can run. Two threads can both read value as, say, 100, both add their amounts (50 and 30), and both write back, one writing 150 and the other 130. Whichever writes last wins, and the other addition is lost. The final value is 150 or 130, never the correct 180. This is exactly the lost-update anomaly from the consistency literature, now in application code rather than a database.
The check-then-act flaw is subtler and the prompt’s mention of summing trades often implies it. A check-then-act is any code that reads a value, makes a decision based on it, and then acts, where another thread can change the value between the check and the act. A trade callback that checks whether the new value would cross a threshold and then emits an alert if it does, can both miss the crossing (if another thread’s update is lost) and double-fire (if two threads both see below the threshold and both cross it). The fix is the same as for the lost update: make the check and the act atomic with respect to other threads, so no other thread can run between them.
A concurrency bug does not live in a line, it lives in the space between a read and a write, where another thread can interleave. To find it, read every read-modify-write and every check-then-act as a family of possible interleavings, and ask which interleaving breaks the invariant. The bug is the interleaving you find, not the line you are staring at.
Visibility and the memory model
The third flaw is the one most candidates miss, because it is invisible in the source. On the JVM, a plain var gives no guarantee that a write by one thread is ever observed by a read on another thread, absent a happens-before relationship. The hardware caches values in per-core caches, the compiler and the JIT are free to reorder operations within a single thread’s semantics, and without a memory barrier (which synchronization primitives provide) a reader can observe a stale value indefinitely.
The practical consequence is that current() can return a value that does not reflect a trade that onTrade already completed, not because of a race window but because the reader’s cache never saw the write. This is a distinct bug from the lost update, and a fix that only addresses atomicity (making the read-modify-write atomic) does not address visibility unless the atomic primitive also establishes a happens-before edge. On the JVM, AtomicLong, synchronized, volatile, and the higher-level concurrency utilities all establish happens-before edges; a plain var does not.
The lesson is that correctness under concurrency has two dimensions, atomicity (no interleaving breaks the invariant) and visibility (writes are observed by reads), and a fix must address both. A candidate who names only atomicity has half the answer.
The fixes side by side
The fixes map to the flaws. The lost update and check-then-act demand atomicity, which on the JVM means either a lock (synchronized or ReentrantLock) or an atomic primitive (AtomicLong with compare-and-set). Visibility demands a happens-before edge, which both locks and atomic primitives provide. The fixed snippet below uses an AtomicLong and a compare-and-set loop, which makes each update both atomic and visible without holding a lock.
The alternative is a lock, which is simpler to write and easier to reason about when the critical section spans multiple operations, at the cost of serializing all updates through the lock.
The tradeoff between the two fixes is contention versus complexity. The atomic compare-and-set is lock-free, so under low contention it is faster and scales as cores are added, but it retries under high contention and its retry loop can waste cycles. The lock serializes all updates, so under high contention it caps throughput at the lock’s hold time, but its critical section is easy to extend to multiple operations (such as updating both a value and a derived cache) without restructuring. The senior judgement is that the atomic fits a single hot counter, and the lock fits a multi-field invariant or a longer critical section.
Immutable snapshots and idempotent appends
Two more fixes solve the problem by removing the mutable shared state entirely. An immutable snapshot replaces the mutable value with a sequence of immutable values, each computed from the previous one. A read receives one immutable snapshot and operates on it, a writer produces a new snapshot by applying the trade to the previous one, and because snapshots are immutable, a reader’s snapshot never changes under it. This is the persistent-data-structure approach, and it trades allocation cost for the elimination of the read-modify-write race entirely.
An idempotent append treats the portfolio value as a derived view of an append-only event log of trades. Each trade is appended to the log keyed by a unique transaction id, and the value is computed by folding the log. A duplicate append with the same id is a no-op, so retries are safe, and the fold is recomputed or incrementally maintained. This is the event-sourcing approach, and it is what a money-facing system actually ships, because it makes the source of truth append-only (which is durable and replayable) and the value a derived projection.
When the value feeds a money-facing display or decision, a mutable shared counter is the wrong model even if synchronized, because it loses history and makes retries unsafe. An append-only log of trades keyed by idempotency id, with the value as a derived fold, is the model a regulated system ships: durable, replayable, and idempotent under retry. The race disappears because there is no mutable shared state to race on.
Testing the fix so it actually holds
A concurrency fix is never proven by a passing test. The best a test can do is raise confidence by maximizing the chance that, if a race remains, the test surfaces it. The testing strategy has four layers, each catching a different class of bug.
The first layer is the stress test. Run many threads performing the concurrent operation in tight loops, with enough total operations that any race with a non-zero probability fires many times. For the portfolio value, that means hundreds of threads each adding known amounts in a loop, then asserting the final value equals the sum of all amounts. A lost update makes the assertion fail, and the failure is reproducible across runs once the contention is high enough.
The second layer is the property-based test. Instead of writing one fixed scenario, describe a property that must hold for any valid sequence of operations, and let the test framework generate random sequences. For the portfolio value, the property is that after any sequence of trades, the value equals the sum of their amounts, and the framework generates random interleavings of concurrent trades to probe it. Property-based tests catch races that a hand-written scenario would miss, because the framework explores far more interleavings than a human would write.
The third layer is the race detector. Kotlin and Java run on the JVM, and the JVM has no built-in race detector, but the companion tools do. For code that shares a JVM with native concurrency analysis, tools like Thread Sanitizer (for native code called via JNI) and static analyzers detect unsynchronized access to shared mutable state. For Go (which the platform-side code uses), the race detector instruments memory accesses at compile time and reports data races at runtime, and running the stress test under it catches races the assertion might miss. A race detector is the strongest tool because it reports the race directly rather than inferring it from a wrong result, but it only sees the interleavings the test actually exercises, so it complements rather than replaces the stress test.
The fourth layer is deterministic reproduction. A flaky test that fails sometimes is hard to fix with confidence, because you cannot tell whether the fix worked or the test got lucky. The discipline is to make the failing test fail reliably, either by raising the contention until the race fires every run, or by using a deterministic concurrency scheduler that controls the interleaving. Once the test fails every time before the fix and passes every time after, you have the strongest evidence short of proof that the fix holds.
Deterministic reproduction and the flaky-test trap
The hardest concurrency bug to fix is the one that fails on the CI run but not locally, or fails only on the third run out of ten, or fails in production but never in any test. This flakiness is the signature of a race condition that is not fully tamed by the test harness, and it erodes confidence because a passing run may mean nothing.
The fix for flakiness is deterministic reproduction. The idea is to control the thread scheduler so that every test run follows the same interleaving, meaning a race either fires every time or never. On the JVM, this is achieved through a model checker or a deterministic scheduler that forces a specific thread to run at each decision point. Libraries like Lincheck (for Kotlin) and JCStress (for Java) provide deterministic concurrency testing by instrumenting the code and exploring all legal interleavings within a bounded scope.
For the portfolio value, a Lincheck test would enumerate every possible interleaving of two concurrent adds with known starting values and verify that the final value is the sum under every interleaving. If any interleaving leaves the value wrong, the test fails deterministically, and the developer sees exactly which interleaving causes the failure. The cost is that the state space explodes with the number of threads and operations, so deterministic testing is bounded to small scenarios (two threads, a handful of operations) and layered with stress testing for larger scenarios.
A candidate who names deterministic reproduction and the tools that provide it (Lincheck, JCStress, or similar) shows they have actually debugged a flaky concurrency test in production, not just read about it.
What changes in a distributed setting
The portfolio-value bug so far assumes a single process with multiple threads. The prompt leaves open whether the value lives in one process or across many service instances. If the value is shared across instances, the fixes change entirely.
A process-local lock does not block a thread in another process from reading or writing the same shared state. An AtomicLong in one process does not make the write visible to a different process. The fix must live at the shared state itself. If the value is stored in a database row, the fix is an optimistic lock (a version column and a conditional UPDATE with retry) for low contention, or a pessimistic lock (SELECT FOR UPDATE) for high contention. If the value is stored in a distributed cache, the fix is a compare-and-swap at the cache layer (Redis provides WATCH and MULTI for this) or a distributed lock through a consensus service like ZooKeeper or etcd.
The most robust distributed fix is the one that removes the mutable shared state entirely: the append-only log approach, where each instance appends trades to a shared log keyed by idempotency id, and the portfolio value is a derived view computed from the log. This is the event-sourcing model, and it is what a regulated brokerage would actually ship, because the log is the immutable source of truth that all instances read and none mutate.
If the portfolio value lives in a shared database accessed by many service instances, a process-local lock or atomic field does nothing to prevent races across instances. The fix must be at the shared state: a row-level lock, an optimistic version check, or an append-only log that eliminates the mutable shared state entirely. Naming this distinction is what separates a senior answer from a junior one.
Common mistakes candidates make
The first mistake is jumping to a fix before forming a hypothesis. Without a hypothesis about which interleaving causes the symptom, the candidate cannot design a test that reproduces the bug, and the fix is a guess. The method is hypothesis first, test second, fix third.
The second mistake is fixing only atomicity and ignoring visibility. A synchronized block or an atomic primitive fixes both, but a fix that only makes the read-modify-write atomic without establishing a happens-before edge (such as a manual lock with the wrong memory semantics) leaves the visibility bug in place. Naming both dimensions is the senior answer.
The third mistake is treating a passing test as proof. A concurrency test that passes once proves almost nothing, because the race may simply not have fired on that run. The test must fail reliably before the fix and pass reliably after, under high contention and many runs, ideally under a race detector.
The fourth mistake is using a process-local lock and assuming it solves a distributed race. This is the same trap as in the distributed-concurrency concept: a lock in one process does not block another. If the portfolio value lives in a shared database accessed by many instances, the fix must live at the shared state (optimistic or pessimistic locking in the database), not in a process mutex.
The fifth mistake is leaving the mutable shared counter in place for a money-facing value. Even correctly synchronized, a mutable counter loses history and makes retries unsafe. The senior answer for money is to move to an append-only log keyed by idempotency id, with the value as a derived fold, which removes the race by removing the mutable shared state.
The sixth mistake is ignoring the flaky-test trap. A test that fails sometimes but not always is not a reliable regression test, and a fix that makes it pass on one run may not have done anything. The discipline of deterministic reproduction, through model checking or a deterministic scheduler, is the senior practice that most candidates omit.
What the interviewer asks next
The first follow-up asks for the specific flaw in the snippet. The expected answer names the non-atomic read-modify-write as the lost update, the check-then-act window if a conditional is present, and the visibility problem from a plain mutable field, and notes that all three must be fixed.
The second follow-up asks how to reproduce the bug. The expected answer is a stress test with many threads in tight loops, asserting the final value equals the exact sum, with a start gate to maximize contention. The test should fail reliably before the fix.
The third follow-up asks how to prove the fix holds. The expected answer is that a fix is never proven, only made more confident, by running the stress test under high contention many times, adding property-based testing to explore more interleavings, and running under a race detector to catch races the assertion might miss.
The fourth follow-up probes the choice between atomic, lock, immutable snapshot, and append-only log. The expected answer is that the atomic fits a single hot counter, the lock fits a multi-field invariant, the immutable snapshot fits a derived projection that must not change under a reader, and the append-only log fits a money-facing value where durability, replay, and idempotent retry matter.
The final probe asks what changes if the value lives in a shared database across instances. The expected answer is that a process-local lock no longer helps, and the fix must be optimistic or pessimistic locking at the database, or an idempotent append with the value derived from the log, which is the event-sourced model a regulated system ships.
Mastery Questions
Question
What is the signature that a portfolio-value bug is a concurrency flaw rather than a logic error, and why?
Answer
The value is occasionally wrong, not consistently wrong, and the error magnitude matches a trade amount. A logic error is wrong every time and at predictable magnitudes; a concurrency flaw is wrong only under specific interleavings, which fire intermittently, and the magnitude reflects whatever update was lost. That intermittence, plus the matching magnitude, points at a lost update in a read-modify-write on the shared value.
Sources & evidence8 claims · 1 cited
Real interview prompt from the Trade Republic interview source; hypothesis-driven debugging, the lost-update and visibility flaw catalog, the fix tradeoffs, and the testing strategy are internal-reasoning and carry no claim.
- The documented Trade Republic debugging prompt describes a user portfolio value that is occasionally wrong after a trade, where the code sums executed trades, and asks the candidate to analyze the snippet for race conditions and logic errors and describe how to test the fix.verified
- The intermittently-wrong signature, with error magnitude matching a single trade amount, points at a lost update in a non-atomic read-modify-write on a shared mutable value rather than a logic error, which would be consistently wrong.internal reasoning
- On the JVM a plain mutable field provides no happens-before guarantee, so a reader may never observe a writer update; correctness under concurrency has two dimensions, atomicity and visibility, and a fix must establish both via a synchronized block, an atomic primitive, or volatile.internal reasoning
- A compare-and-set loop on an AtomicLong is lock-free and fits a single hot counter; a synchronized block serializes updates but fits a multi-field critical section; an immutable snapshot eliminates the race for readers; an append-only log keyed by idempotency id removes the mutable shared state and is the model a money-facing system ships.internal reasoning
- A concurrency fix is never proven by a passing test; confidence scales with the interleavings explored, operations per run, and runs, via stress tests with many threads and a start gate, property-based tests, and a race detector, with deterministic reproduction so the test fails reliably before the fix.internal reasoning
- An append-only event log keyed by idempotency id eliminates the portfolio-value race by removing mutable shared state, making the value a derived fold over an immutable log.internal reasoning
- Debugging a concurrent portfolio-value race condition where the sum of executed trades is occasionally wrong is a real Trade Republic interview prompt used in the onsite coding round.verified
- The lost-update race in a non-atomic read-modify-write requires fixing both atomicity and the happens-before edge (visibility) to be fully correct.internal reasoning
Cited sources
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)