On this path · System Design 6. Putting It Together: A Senior's End-to-End Mental Model
Putting It Together: A Senior's End-to-End Mental Model
Synthesizing the path from a product tap to a durable, regulated, real-time execution.
Learning outcomes
Every concept you have studied in this knowledge base is a piece of a single machine. The matching engine, the outbox, the change data capture pipeline, the Redis fan-out, the ledger, the regulatory frame: none of them exists in isolation. A Senior engineer at Trade Republic is paid to hold the whole machine in their head and reason about how a tap on a phone becomes a durable, regulated, real-time execution. This capstone traces that journey end to end and shows how each layer composes.
After studying this page, you can:
- Walk one customer journey, from a buy tap to a settled trade, and name the system responsible at every hop.
- Justify, at each stage, why the design is what it is, which alternatives were rejected, and what breaks under load.
- Explain how the documented failure modes (unhandled transaction rollbacks, assumed exchange format consistency, monolithic ingestion, self-centric motivation) map onto real stages and how a senior avoids each.
- Defend the composition as a Staff interviewer probes the boundaries between teams.
- Articulate the operating model that lets a Senior engineer own incidents that cross team lines.
The page is deliberately structured as a walkthrough rather than a reference. You will not find a flat catalog of services. You will find one trade moving through ten stages, with the engineering judgment surfaced at each hop. Read it once end to end to build the map, then return to any single stage when you need to sharpen your defense of that piece.
Before we dive in
A customer opens the app, taps a button to buy ten shares, and a fraction of a second later the position shows up on the screen. From the outside it feels like one action. From the inside it is ten coordinated stages that span a mobile client, a regulated order service, a double-entry ledger, a matching engine, a transactional outbox, a change data capture pipeline, a stream processor, a real-time fan-out, a settlement batch, and an observability stack.
No single person designed this whole chain, and that is the first lesson of the capstone. The machine is the composition of many local designs, each owned by a different team, each defending its own invariant. The Senior engineer’s job is not to know every line of every service. It is to know the invariant each stage guarantees, the contract one stage hands to the next, and the place where a failure in one stage becomes a silent corruption in another.
The reason this page exists is that the failure modes that sink candidates and incidents are almost never inside a single stage. They live in the seams. A trade that commits in the ledger but never reaches Kafka. A matching engine that assumes every exchange returns the same JSON shape. An ingestion service that was built as one monolith because that felt simpler, and now cannot scale. A candidate who, when asked why they want the job, talks about their career growth instead of the mission that funds the company. Each of these is a seam failure, and each has a stage in this walkthrough where a senior either catches it or lets it through.
The structure of the walkthrough mirrors how a senior debugs an incident, not how a textbook presents a topic. We do not start with the database and build up. We start with the customer’s finger on the screen, because that is where the requirement originates, and we follow the consequence of that tap downstream until the shares are settled and the position is back on the screen. At each hop we pause and ask the engineering question: what does this stage promise the next one, what can go wrong, and what would I, as the on-call owner, be paged for. That is the mental model a Senior engineer carries, and it is built hop by hop, not from a diagram memorized whole.
The story we will trace is one trade. The thread that runs through every stage is the same: convert a distributed coordination problem into a local invariant plus an operational contract, and make that contract observable. That is the whole craft.
Hold that thread as you read. Every stage that follows is an instance of it, and the capstone’s payoff is seeing the same move repeated in ten different forms until it becomes instinct.
The whole flow at a glance
Before we descend into any single stage, you need the map. The diagram below is the centerpiece of this capstone. Read it left to right as the forward path of the order, and read the lower arc as the return path of the real-time update. The two paths are not symmetric: the forward path is built for durability and correctness, the return path is built for latency and fan-out.
The arc to hold in your head is this: the forward path never talks to the return path directly. The order service does not push a notification to the client. It only writes a ledger row and an outbox row in one transaction, and then the change data capture pipeline takes over. That separation is the entire reason the system is reliable. The order service stays simple and fast because it does one transactional thing. The return path is eventually consistent and can be replayed, because it is fed from a durable log.
This separation has a cost that a senior states honestly rather than hiding. Because the return path is fed from the log and not from a direct push, there is latency between the commit and the client seeing the update. That latency is bounded by the Debezium capture delay, the Kafka consumer lag, and the stream processing time, and in normal operation it is well under a second. But it is not zero, and a senior never claims it is. The customer’s screen updates a heartbeat after the trade, not instantaneously, and the optimistic UI on the client bridges that heartbeat by showing the expected result immediately and correcting only if the server disagrees. The honest statement of this latency, and the design that absorbs it, is what makes the eventual consistency trustworthy rather than surprising.
Every stage in this diagram owns one invariant and exposes one contract. The Senior engineer trusts the upstream contract and defends the downstream one. When you can name the invariant and the contract at each hop, you can reason about the whole machine without reading all of it.
There is a second principle the diagram encodes, and it is the one a Staff interviewer will press on. Notice that the forward path and the return path meet at exactly one place: the transactional write of the ledger row plus the outbox row. That single commit is the hinge of the whole system. Everything before it is about getting a correct trade into the database. Everything after it is derived from the database through the log. The hinge is narrow on purpose. A narrow hinge means there is one place to reason about consistency, one place to test, and one place to audit. A wide hinge, where the forward path reaches into many downstream systems directly, scatters the consistency question across the whole codebase and makes it unanswerable. A senior designs for a narrow hinge even when a wider one would save a few lines of code, because the narrow hinge is what keeps the system comprehensible as it grows.
Now we descend into each stage. For every stage we answer the same five questions: what does it do, why is it designed this way, what was the rejected alternative, what breaks at scale, and what will a Staff interviewer probe.
Stage 1: The tap and the mobile-first contract
The journey starts in the customer’s hand. Trade Republic ships native iOS in Swift and Android in Kotlin with Jetpack Compose, and the mobile client is a first-class product surface, not a thin shell over a web app. The tap to buy ten shares is a single user gesture that becomes an order intent: an authenticated HTTPS request carrying the instrument identifier, the quantity, and an idempotency key.
The design choice that matters here is mobile-first as an engineering principle, not just a design principle. The client owns retry, optimistic UI, and idempotency, because the mobile network is lossy and the customer will tap twice when the train goes through a tunnel. The order intent carries an idempotency key so that a duplicate tap does not become a duplicate order. This is the first place a senior looks when a customer complains that they were charged twice: not at the backend, but at whether the client generated a fresh key on each retry.
The contract the client hands the backend is precise: an idempotent order intent, authenticated, with a client-generated key. The contract the backend hands the client is vaguer but just as important: the server is the source of truth for whether the order happened, and the client may resynchronize its view at any time without ambiguity. A senior writes both contracts down, because the seam between mobile and backend is exactly where “works on my machine” failures hide.
The rejected alternative is a server that deduplicates on business state alone, without a client supplied key. That fails because the server cannot distinguish a genuine second order from a retried first one. The idempotency key turns a fuzzy business question into a precise mechanical one.
What breaks at scale is the assumption that the network is reliable. It is not, and a senior assumes the opposite: every request may be delivered zero, one, or many times, and the system must be correct under all three. The interviewer probes this by asking what happens if the customer closes the app between the tap and the response. The defensible answer is that the order lives or dies on the server, never on the client, and the client resynchronizes its view from the return path described in stage 8.
Stage 2: Order validation against the regulatory frame
The order intent reaches the order service through an API gateway. The first thing the service does is not check the market, it checks the law. Trade Republic operates inside a regulatory frame that demands, before any order is accepted, that the customer has passed Know Your Customer checks, has signed the appropriate suitability documents under MiFID II, and is not exceeding position or concentration limits.
The design choice is to fail closed. If any regulatory check cannot be answered with a definitive yes, the order is rejected. The reason is asymmetric risk: a wrongly accepted order can create a regulatory liability that costs orders of magnitude more than a wrongly rejected one. A senior treats regulatory checks as a hard gate that precedes every other concern, because no performance optimization is worth a fine or a license review.
The regulatory frame is not a single check but a stack, and a senior can name the layers. Identity verification confirms the customer is who they claim and has passed anti-money-laundering screening. Suitability confirms the customer understands the product class they are buying, which for complex instruments like derivatives or bonds demands a documented assessment. Limits confirm the order does not breach per-position or portfolio concentration rules the firm sets to protect the customer and itself. Each layer is independent, each can veto the order, and each has its own source of truth and its own staleness profile. The senior models them as a chain of gates, not a single boolean, because a failure in any one gate must block the order even if the others pass.
The rejected alternative is to run regulatory checks asynchronously after the order is accepted, to keep the latency of the accept path low. That trades a few milliseconds of perceived latency for a class of bug where a customer trades outside their permitted limits and the firm finds out later. The latency saving is not worth the risk, so the check is synchronous and on the critical path.
What breaks at scale is staleness. A regulatory state that is cached too aggressively can let a customer trade after their limits have been tightened. The senior defense is to bound the cache by a short time to live and to design the limit service so that a limit change invalidates the cache proactively. The interviewer probes this by changing a limit mid-session and asking whether the next order is blocked. The correct system blocks it, even if the cache has not expired, because the limit write invalidates the cache before it returns.
A subtlety worth naming is the difference between a check that fails open and one that fails closed, and why a brokerage chooses closed. A fails-open system treats an unreachable dependency as permissive: if the limit service is down, let the order through and reconcile later. A fails-closed system treats the same outage as restrictive: if the limit service is down, reject the order. The first maximizes availability, the second maximizes correctness. In a consumer broker where every wrongful trade is a regulatory event, the senior picks closed and tells the interviewer why, in plain language, the availability cost is acceptable. This is a judgment call, not a technical fact, and stating it explicitly is what separates a senior from someone reciting best practices.
Stage 3: Risk and balance checks over the ledger
Once the order clears regulation, the service must confirm the customer can pay for ten shares. This means reading the customer’s cash balance and current holdings, and confirming the purchase does not breach margin or risk limits. The source of truth for both is the double-entry ledger in PostgreSQL, where every movement of cash or stock is recorded as a balanced pair of entries.
The design choice is to read balances from the ledger, not from a cached balance service, for the authoritative risk decision. A cached balance is fine for display, but the risk check must see the ledger’s committed truth. The reason is that the ledger is the only place where a balance is provably consistent with every prior movement. Reading anywhere else introduces a window in which a customer could spend money they no longer have.
A useful way to hold this is to distinguish three kinds of reads in the system, because confusing them is the root of most money-safety bugs. A display read is approximate and may lag; the screen uses it. A risk read must be authoritative and recent; the order path uses it. A settlement read must be exact and historical; the reconciliation uses it. A senior never lets a display read drive a risk decision, because a stale-but-fast balance is exactly the wrong input for a decision that can move money. The three reads have different sources, different freshness contracts, and different consumers, and keeping them straight is a daily discipline, not a one-time design.
The rejected alternative is to keep a denormalized balance column on the customer table and update it in place. That is faster to read but loses the audit trail that a regulated broker is required to keep, and it makes a partial failure invisible. Double-entry is slower and more verbose on purpose, because every entry has a matching opposite entry and the books always balance. The verbosity is the safety.
A senior can defend the verbosity with a single argument: in a double-entry ledger, the invariant that total debits equal total credits is enforced by construction, not by a later reconciliation that might pass or fail. That means an inconsistency is not a quiet drift discovered months later; it is a write that the database rejects at commit time. The cost is two writes per movement instead of one. The benefit is that the books cannot silently unbalance, which in a regulated firm is the difference between a clean audit and an enforcement action. The senior accepts the cost because the invariant it buys is the one the regulator cares about most.
What breaks at scale is concurrency. Two orders for the same customer can race, each reading the same balance and each deciding the customer has enough funds, and together they overdraw the account. The defense is to serialize conflicting writes, either through selective row locks on the customer’s balance row or through a constraint that rejects the second write, never through optimistic assumptions. A senior never assumes the read balance will still be valid at commit time, because in a brokerage that assumption is exactly where money leaks.
The interviewer probes this by asking what happens when two devices place orders at the same instant. The correct answer names the locking or constraint strategy explicitly and admits the cost: the second order waits or fails, and the system is correct, not fast and wrong.
There is one more design choice at this stage that a senior defends carefully: the ledger is the system of record, but the risk decision is made on a read at order time, not at commit time. Between the read and the commit, the balance can change. The defense is to re-check the invariant inside the committing transaction, so that the risk decision that mattered was the one taken under the same lock that wrote the row, not the one taken milliseconds earlier on a stale read. A senior never lets a read-decide-commit sequence span two transactions without a guarantee that the decision still holds at commit, because that gap is precisely where two racing orders both pass and together overdraw the account.
Stage 4: Routing and matching by price-time priority
With funds confirmed, the order is routed to the matching engine. For a retail broker, many orders are routed to external venues, but the mental model is the same: the order meets the market and is matched by price-time priority. A better price wins, and at the same price the earlier order wins.
The design choice that matters at this stage is that the matching engine is treated as a deterministic function of an ordered input stream. The engine does not query balances, does not write the ledger, and does not notify the client. It receives orders in a defined sequence and emits fills in a defined sequence. Keeping the engine pure is what lets it be fast and testable, and it is what lets the rest of the system trust its output.
The rejected alternative is the monolithic engine that matches, updates balances, and publishes notifications in one call. That feels simpler on a whiteboard and is the documented trap of building one service that does everything. The monolith cannot scale the hot path independently of the cold path, cannot be replayed for testing, and turns every change into a cross-cutting risk. The senior defense is to keep the engine pure and push every side effect into the transactional stage that follows.
What breaks at scale is ordering ambiguity under partitioning. If two matching instances can both claim the same resting order, they can double-fill it. The defense is to partition by instrument identifier so that all orders for one instrument land on one engine instance, and to make the partitioning deterministic so that a failover resumes from the same assignment. The interviewer probes this by asking what happens if two engines briefly believe they own the same instrument. The correct answer is that the system is designed so that cannot happen for long, and that the recovery reconciles from the durable log rather than from engine memory.
The reason the engine must be deterministic runs deeper than performance. A deterministic engine is replayable: given the same ordered input, it produces the same ordered output. That replayability is what lets the team test the engine against a week of historical orders, what lets them rebuild state after a crash from the log, and what lets them prove, in an incident review, that a controversial fill was the only correct fill given the inputs. A non-deterministic engine, one that reads the clock or consults a random source during matching, cannot be replayed and therefore cannot be audited. In a regulated broker, an unauditable matching path is a compliance failure, not just an engineering one. The senior states this trade-off out loud: determinism costs a little flexibility and buys auditability, and in this domain auditability wins.
Stage 5: The atomic ledger write plus outbox row
The fill comes back to the order service. Now the service must do three things: update the ledger to reflect the new holdings and the spent cash, record that the trade happened durably, and notify the downstream world. The whole reliability story of the system turns on how these three are combined.
The design choice is the outbox pattern. Instead of writing the ledger and then publishing to Kafka in two separate calls, the service writes the ledger rows and an outbox row inside one PostgreSQL transaction. Either all three commit, or none do. There is no window in which the ledger is updated but the event is missing, and no window in which the event exists but the ledger does not. The dual-write problem is dissolved by collapsing two writes into one transaction.
Kafka does not participate in PostgreSQL transactions. A publish is a network side effect that the database cannot roll back, so a rollback after the publish leaves a phantom event, and holding the transaction open for the network call creates a long-running transaction that blocks vacuum. The outbox removes the publish from the transaction entirely.
The rejected alternatives are the three naive attempts. Publish first then commit risks phantom events: downstream systems react to a trade the database later rolls back. Commit first then publish risks lost events: a crash after commit means the trade is permanent but the event never fires. Publish and commit in one transaction fails because the publish is a side effect the database cannot undo. Each attempt is a documented failure mode, and the outbox exists because all three are unsafe.
What breaks at scale is everything that happens after the pattern is in place. The outbox table is a high-churn table, and the publisher that drains it is an operational system in its own right. The documented incidents at this stage include clock skew breaking ordering, integer sequences overflowing, the wrong index collapsing query times, large batches blocking autovacuum, poison pills stalling the whole publish transaction, and visibility-map decay forcing a partitioned redesign. A senior does not treat the outbox as a pattern that is done once written; it is a subsystem that earns its own on-call attention.
The interviewer probes this by asking, after the commit, whether the event is guaranteed to reach Kafka. The honest answer is yes, eventually, because the outbox row is durable and a publisher will drain it, but the delivery is at-least-once, not exactly-once, which is why every downstream consumer must be idempotent. The senior states the delivery contract out loud instead of letting the interviewer extract it.
A note on the transaction itself, because this is where the documented failure mode of unhandled rollbacks lives. The transaction can fail at commit, and when it does, neither the ledger row nor the outbox row is written, because they share one transaction. That is the safe case: a rollback undoes both atomically, and the customer sees the order as not executed. The dangerous case is the design that writes the ledger, commits, and then publishes as a separate step. There, a failure between commit and publish leaves the ledger updated and the event missing, and a retry of the whole flow debits the customer twice. The outbox design makes the rollback free of side effects precisely because there is no side effect to undo. The principle a senior internalizes is that a rollback must leave the world exactly as it was before the transaction started, and the only way to guarantee that across a system boundary is to make the boundary-crossing action durable-with-the-data rather than performed-after-the-data.
Stage 6: Debezium streams the outbox row from the WAL to Kafka
The outbox row is now durably committed, but Kafka does not know about it yet. The bridge is change data capture. Debezium connects to PostgreSQL through logical replication and reads the write-ahead log. Every committed change to the outbox table appears in the WAL, and Debezium ships it to Kafka as an event, with no extra query load on the database and no polling.
The design choice is CDC over polling for two reasons. First, polling adds redundant database load, a latency versus load tradeoff, and operational complexity, all to rediscover changes the database already knows about. Second, the WAL is the authoritative record of what committed and when, so reading the WAL gives both low latency and correct ordering without any application-level bookkeeping. Debezium turns the database into the source of an event stream.
The serialization choice inside this stage is Avro, and the specific Trade Republic design serializes the outbox payload to bytes on the application side before insertion, storing it as a BYTEA column. The reason is fail-fast schema handling. If a producer emits a payload that does not match the registered schema, the failure happens at insert time in the producer, before it ever reaches Kafka. Debezium passes the bytes through untouched, which means Debezium itself can never produce a poison pill from a schema mismatch, because it never parses the payload. The schema is managed explicitly through a registry, not discovered at runtime.
The rejected alternative is the polling publisher that queries the outbox table every few seconds for unpublished rows. Polling works at low volume, but at Trade Republic’s scale it adds redundant load, introduces a latency floor equal to the poll interval, and creates a long-running transaction risk when the batch is large. CDC has none of these costs because it reads the log the database writes anyway.
What breaks at scale is the assumption that the connector is fire and forget. A Debezium connector that falls behind grows its WAL retention on the source database, which can fill the disk and stall the primary. The senior defense is to monitor connector lag as a first-class metric and to alert on WAL retention growth, because a slow consumer of the log is a database outage in slow motion. The interviewer probes this by asking what happens if Kafka is down for an hour. The correct answer is that the WAL retains the changes, Debezium catches up when Kafka returns, and the database disk must be sized to absorb that backlog.
A second, subtler breakage is schema evolution. The Avro contract that made fail-fast insertion safe at stage 5 also has to evolve: new fields appear, old fields retire, and every consumer must keep reading. The senior defense is to manage the schema through the registry with backward-compatible evolution rules (new fields are optional, old fields are never removed violently), and to treat a breaking schema change as a coordinated release across producers and consumers, not a surprise. The poison pill that the stage 5 design avoided at insertion time can still appear at consumption time if a consumer cannot deserialize a new schema version, so the consumers are versioned defensively and the registry enforces compatibility before a new schema is accepted. The schema is part of the contract between stages, and a senior guards it as carefully as the database transaction.
Stage 7: Kafka Streams build derived state and sink to Redis
The trade-executed event is now on Kafka. From here the system branches. One branch is cold: the event flows to an analytics warehouse for reporting and regulatory reporting. The branch we follow is hot, because it is what gets the update back to the customer’s screen.
The hot branch is a Kafka Streams topology. The stream processor consumes the raw trade events, joins them with the customer’s prior positions and the latest market quotes, and computes derived state: the updated portfolio value, the real-time yield to maturity on bond holdings, the realized and unrealized gains. The output is sunk to a Redis cluster that acts as the hot read model for the client facing services.
Worth separating clearly is the cold branch that runs alongside the hot one. The same trade-executed event that feeds the stream processor also flows to an analytics warehouse, where it becomes the substrate for regulatory reporting, customer statements, and business analytics. The two branches read the same log but answer different questions. The hot branch answers “what does this customer see right now,” and the cold branch answers “what did we do this quarter, and can we prove it to a regulator.” A senior keeps them separate because their latency, correctness, and retention requirements are opposed: the hot branch favors low latency and eventual consistency, the cold branch favors completeness and exact historical accuracy. Trying to serve both from one path compromises both.
The design choice worth naming is why this work lives in a stream processor and not in the order service. The order service’s job is to commit the trade correctly. Computing a portfolio value requires joining the trade against positions, quotes, and reference data, which is a different concern with different latency and scaling characteristics. Putting that work in the order service would re-introduce the monolith: the hot trading path would now also carry the weight of the analytics path, and a slow quote join would stall an order commit. The stream processor isolates the derived computation so that each path scales to its own demand.
The sink to Redis uses sharded pub and sub, not the plain pub and sub. The reason is specific and documented. Plain Redis pub and sub broadcasts every message to every node in the cluster, which means every node carries the full firehose regardless of which subscriptions it holds. At millions of subscriptions this destroys scalability because the cluster’s capacity is bounded by the node with the most subscriptions, and every node pays the full cost. Sharded pub and sub routes a message only to the node that owns the channel’s hash slot, so the load is distributed across the cluster in proportion to where the subscriptions actually live.
Plain pub and sub is simpler and fine at low fan-out. It becomes a bottleneck exactly when fan-out is high, which is the only time you reach for it. Sharded pub and sub pays a small routing cost to keep the cluster’s capacity growing with the cluster size, not pinned to one node.
The rejected alternative is to compute the portfolio value on read, by querying the ledger and the quote service every time the client opens the app. That works for a customer with one position who opens the app once a day. It collapses at scale, because every screen open becomes a fan-out of reads against the ledger and the quote service, and a market move triggers every customer to refresh at once. The precomputed hot read model absorbs that read amplification by computing once and serving many.
The interview question that separates a senior here is about consistency between the hot read model and the ledger. The hot read model is a derived view, and it can lag the ledger by the CDC and stream-processing delay. A customer who just traded might see their old position for a heartbeat before the update arrives. The senior does not pretend the two are always in sync. The senior designs the client to treat the hot read model as the display layer and the ledger as the authoritative layer, and to reconcile them on every meaningful action. When the customer taps to sell, the risk check reads the ledger, not the display, so a stale display can never authorize a trade the customer cannot afford. The display is allowed to lag; the decision is never allowed to.
What breaks at scale is state store sizing and rebalancing. A Kafka Streams state store holds the materialized portfolio per customer, and when the stream processor rebalances, the new instance must rebuild or transfer that state. The senior defense is to size the state stores for the customer base, to use persistent storage local to the stream instance, and to design the topology so a rebalance is a recovery, not a recalculation from scratch. The interviewer probes this by asking how long a new stream instance takes to become ready. The honest answer names the standby replica strategy that keeps a warm copy so failover is seconds, not minutes.
One more judgment call lives here: where to draw the line between compute-on-write and compute-on-read. The portfolio value is computed in the stream because it is read by every customer on every screen open, and recomputing it per read would amplify a market move into a read storm. But not every derived value belongs in the stream. A value that is rarely read, or that depends on data the stream does not have, is better computed on read against the ledger. The senior does not put everything in the stream processor out of dogma. The senior asks, for each derived value, how often it is read, how expensive it is to compute, and how stale it may be, and routes accordingly. The stream processor is the answer for the hot, frequently-read, freshness-sensitive values; the ledger is the answer for the cold, rarely-read, must-be-exact values.
Stage 8: Ktor WebSockets stream the update back to the client
The derived update is now in Redis. The last hop of the return path is the WebSocket service that pushes it to the customer’s phone. Trade Republic’s documented design for real-time streaming uses Ktor WebSockets on the server and a Kotlin SharedFlow on each service instance to fan one instrument’s updates to every connected client subscribed to it.
The design choice is the SharedFlow fan-out combined with on-demand Redis subscriptions. When the first client subscribes to an instrument, the service creates one Redis subscription for that instrument. When the last client unsubscribes, the service tears that subscription down. Between those two events, one Redis subscription feeds a SharedFlow that broadcasts to every connected WebSocket for that instrument. The resource cost is one Redis connection per service instance regardless of how many clients are connected, which is what makes the design scale.
The rejected alternative is one Redis subscription per client. That feels like the obvious mapping: each client wants updates, so each client gets its own subscription. It fails because the number of Redis connections grows with the number of clients, and a market move that updates a popular instrument fans the same message across thousands of duplicate subscriptions. The SharedFlow design collapses those duplicates into one subscription and one broadcast, trading a small amount of in-process fan-out work for a large reduction in Redis connections.
What breaks at scale is backpressure. A slow client, or a client on a degraded network, cannot drain its WebSocket fast enough, and the SharedFlow buffer can drop updates or block the emitter. The senior defense is to size the buffer for the expected update rate, to define what the client sees when an update is dropped (typically a full resynchronization from the latest snapshot), and to never let one slow client stall the broadcast to the rest. The interviewer probes this by asking what the client sees if it falls behind. The correct answer is that the client resynchronizes from the current state, not from a gap-filled replay, because correctness in finance means showing the latest truth, not the history of how you got there.
The backpressure choice is a deliberate trade-off between three undesirable outcomes, and the senior names all three rather than pretending there is a free lunch. The first option is to buffer unboundedly, which protects against drops but lets one slow client consume unbounded memory and eventually crashes the service. The second is to drop updates for the slow client, which protects the service and the other clients but leaves the slow client stale until it resynchronizes. The third is to block the emitter, which protects neither the service nor the other clients and lets one slow connection slow down everyone. The senior picks the second, because in a fan-out the many must not suffer for the one, and because staleness is recoverable while a service crash is not. The resync-on-drop contract is what makes that choice safe: the client always has a way back to correctness without the server holding its history.
There is also a connection-lifecycle concern that a senior defends. WebSockets are long-lived connections, and a service restart that drops every connection at once causes a thundering herd of reconnects. The defense is client-side jitter on reconnect and server-side graceful shutdown that drains existing connections before exiting. The interviewer probes this by asking what happens when the WebSocket service deploys. The correct answer is that existing clients stay connected until drained, new clients route to the new version, and no customer sees a frozen position because of a deploy. A deploy that visibly breaks the real-time stream is an operational bug, not a normal cost of shipping.
Stage 9: Settlement at T plus 1 and batch reconciliation
The trade is matched, the ledger is updated, the client sees the new position, and the journey is still not over. The shares and the cash must actually move between broker and custodian, and that movement happens on a settlement cycle, typically T plus 1, one business day after the trade. The real-time system we have built so far is the front office truth. Settlement is the back office truth, and the two must agree.
The design choice at this stage is a batch reconciliation architecture running on Kubernetes. Trade Republic’s documented approach is to scale batch jobs horizontally as partitioned workers on Kubernetes, rather than to build one monolithic ingestion or settlement service. Each worker owns a disjoint slice of the work, identified by a partition key, and processes its slice independently. The batch layer is separate from the real-time layer because its concerns are different: settlement is not latency sensitive, but it is correctness sensitive and reconciliation heavy.
The rejected alternative is the monolithic ingestion service that pulls the custodian file, parses it, reconciles it, and writes the ledger all in one process. This is one of the documented failure modes. The monolith cannot scale: if the custodian file grows, the one process runs longer, and if it fails halfway, the whole batch must restart. The partitioned design means a larger file just spawns more workers, and a failed worker restarts only its slice.
The reason partitioning works here, and fails elsewhere, comes down to whether the work is embarrassingly parallel. Settlement reconciliation per customer is independent of every other customer: worker A reconciling customer 1 does not need to see worker B reconciling customer 2. That independence is what lets the batch scale horizontally with no coordination. A senior recognizes this property and exploits it, partitioning by customer id range so each worker owns a disjoint slice. The same property does not hold for the matching engine, where orders for one instrument must be serialized, which is why the two systems scale differently despite both being parallelizable in spirit. Naming the difference, rather than applying one scaling recipe everywhere, is the senior’s contribution.
What breaks at scale is the assumption that every custodian and every exchange returns data in a consistent format. This is the second documented failure mode, and it is seductive because in a clean room every feed looks the same. In production, every venue has its own quirks: date formats, decimal separators, identifier schemes, and field presence all vary. The senior defense is to parse defensively, to treat every field as untrusted until validated, and to route unparseable records to a breaks queue for human review rather than guessing. A senior never writes the reconciliation assuming the format will not change, because formats change the moment a venue upgrades its system, and that change is never announced in advance.
The interviewer probes this by handing you a reconciliation where the cash on the custodian file does not match the ledger. The senior does not reach for an automatic write to force agreement. The senior asks which side is wrong, routes the break to review, and treats the mismatch as a signal to investigate, because forcing agreement silently is how real money disappears.
A senior also asks how the breaks queue is drained, because a queue that nobody works is just a slower way of losing money. The operational answer is that breaks have a service-level expectation: a break is reviewed within a target number of hours, aged breaks escalate, and the root cause of recurring breaks feeds back into the parser as a new validation rule. The reconciliation is not finished when the workers stop. It is finished when the breaks queue is empty and the rules that would have caught the break are in place for the next run.
Stage 10: Observability and cross-team incident ownership
Every stage so far has been a design. This stage is what makes all of them operable. Grafana LGTM, the unified stack for logs, metrics, traces, and profiles, watches every hop from the order service through the matching engine, through Debezium, through the Kafka Streams topology, to the WebSocket fan-out. Without this layer, the composition we have built is a black box that works until it does not, and then nobody knows why.
The design choice at this stage is end-to-end observability as a first-class engineering concern, not an afterthought. A request that crosses ten services must carry a trace context that survives every hop, so that when a customer reports that their position did not update, an on-call engineer can reconstruct the exact path the order took and see which stage dropped it. The metrics that matter are not generic: they are connector lag on Debezium, consumer lag on Kafka, fan-out buffer drop rate on the WebSocket service, and reconciliation break count on the settlement batch. Each metric is tied to a specific failure mode of a specific stage.
A good metric is not a number on a dashboard. It is a named failure mode you have thought about in advance, attached to a threshold that means “investigate now” and an owner who knows what to do when it fires. A connector lag metric without a threshold is decoration; a connector lag metric with a threshold and a runbook linked to it is the thing that turns a 3am WAL-full outage into a 3am restart.
The discipline that makes this work is the alert budget. A service that pages on every jitter generates noise, and noise trains on-call engineers to ignore pages, which means the real incident is missed when it arrives. A senior treats every alert as a cost, not a benefit, and prunes alerts that fire without indicating action. The target is a small number of high-signal alerts, each tied to a stage’s invariant, each with a clear first response. An alert that nobody knows how to act on is worse than no alert, because it consumes attention without producing a fix.
The rejected alternative is per-team observability, where each team instruments its own service and stops at its boundary. That fails exactly when it is needed most, at the seam between two teams. A position that does not update might be a ledger bug owned by the ledger team, a CDC lag owned by the platform team, or a WebSocket drop owned by the client team. Without a cross-team trace, each team can honestly say their service looks healthy while the customer’s position is still wrong.
The span that the senior cares about most is the one that crosses a team boundary, because that span is where ownership is ambiguous. A trace that stops at the order service and resumes at the stream processor, with a gap in between, is useless during an incident, because the gap is exactly where the bug usually is. The senior defense is to make trace propagation a non-negotiable contract at every service boundary, owned by the platform team and enforced in the Tech Radar, so that no team can ship a service that drops the trace context. The cost is a little extra discipline on every new service. The benefit is that when something breaks at 3am, the on-call engineer sees one continuous trace and knows where to look first.
What breaks at scale is the blast radius of an incident. A Debezium lag incident does not stay in the platform team; it manifests as stale positions for every customer, which the client team gets paged for. The senior operating model at Trade Republic, supported by the architecture forum of Senior and Senior-plus engineers and the Tech Radar that standardizes choices across autonomous teams, is designed for exactly this. The on-call senior owns the incident across team boundaries, not just their team’s piece of it, and drives the coordination that resolves it. Ownership does not stop at the service edge.
Where a senior earns the title
Everything so far is engineering. This section is about the judgment that separates a Senior engineer from a strong mid-level one, and it is where the interview is ultimately decided. The capstone would be incomplete without naming the four documented failure modes explicitly and showing how a senior avoids each, because these are the modes that recur across every stage we walked through.
The first mode is not handling transaction rollbacks on failure. We met this at stage 5. A naive implementation publishes the event inside the transaction, the transaction rolls back, and a phantom event drives downstream systems to act on a trade that never happened. A senior never lets a side effect live inside a transaction that can roll back. The senior writes the side effect as a row in the same transaction, so the rollback takes the row with it, and delegates the actual publish to a process that reads committed rows. The principle generalizes: anything that crosses a system boundary must be made durable before it is acted on, never acted on speculatively inside a transaction that might undo.
The second mode is assuming all exchanges provide consistent data formats. We met this at stage 9. A naive reconciliation parser assumes the custodian file looks like the spec, and the day the spec drifts, the parser silently corrupts the ledger or crashes the whole batch. A senior treats every external feed as adversarial until proven otherwise, parses defensively, validates every field, and routes the unparseable to review. The principle generalizes: at every boundary to an external system, assume the format will change without notice and design the parser to fail safe rather than fail corrupt.
The third mode is building a monolithic ingestion service instead of a horizontally scalable one. We met this at stages 4, 7, and 9, because the monolith temptation recurs everywhere. The matching engine that also updates balances, the order service that also computes portfolio value, the settlement job that does everything in one process. Each feels simpler on a whiteboard and each collapses at scale because the hot path cannot be scaled independently of the cold path. A senior decomposes by concern and scales each concern to its own demand, accepting more services in exchange for independent scaling and clearer ownership.
The fourth mode is the one that is not technical at all, and it ends more interviews than the other three combined. It is focusing on personal career growth over the business mission. Trade Republic exists to democratize investing, and the interview grades whether a candidate distills complex financial products into simple mobile-first experiences with regulatory awareness and scalable design. A candidate who, when asked why they want the job, talks about their own growth has misunderstood the role. A senior talks about the customer who could not invest before and now can, about the regulatory care that keeps that customer safe, and about the scale that makes the product reach millions. The technical depth we walked through is necessary but not sufficient. The mission alignment is what makes the technical depth worth hiring.
The defense against this mode is not a rehearsed answer. It is a genuine orientation that shows up in every technical choice the candidate makes. The candidate who frames the outbox as “protecting the customer’s money” instead of “an interesting distributed systems problem” is signaling the orientation the firm hires for. The candidate who frames the regulatory gate as “keeping the customer out of trouble” instead of “a compliance annoyance” is doing the same. The mission is not a separate behavioral round; it is the why behind every technical decision, and a senior weaves it into the defense of the design rather than saving it for a different question.
A senior can defend every design choice as an interviewer probes it. That defense is not memorized. It comes from having held each invariant in turn and asked, at every seam, what happens if the upstream contract breaks. Practice the walkthrough by tracing one trade and, at each stage, naming the invariant, the contract handed downstream, and the failure mode if the contract is violated.
To make the cross-team ownership concrete, picture an incident that the capstone is built to diagnose. At 9:14am, customers report that their portfolio values are frozen, even though the market is moving and their orders are filling. The on-call senior is paged. A mid-level engineer checks the order service and sees it healthy: orders are committing, latencies are normal. A mid-level engineer checks the WebSocket service and sees it healthy: clients are connected, no drops. Each team can honestly say their service is fine. The senior engineer looks at the cross-team trace and notices that the trace context dies between the order service commit and the stream processor. That gap is the Debezium connector, and the metric that confirms it is connector lag: Debezium is hours behind, the WAL retention on the primary is climbing, and the stream processor has not received a trade-executed event since 7:40am. The fix is on the platform team’s side (restart and resize the connector), but the incident was owned by the senior who read the trace across all three teams’ boundaries and refused to declare “not mine” at any of them.
That is what the title is awarded for. The senior did not need to know the Debezium source code. The senior needed to know the invariant each stage owns, the contract between stages, and the willingness to follow the trace across the seam until the gap is found. The architecture forum, the Tech Radar, and the RFC and ADR process exist to make those invariants and contracts explicit and shared, so that during an incident no senior has to guess what the team next door promised. The operating model is the social substrate that makes the technical composition work under pressure.
The throughline of the whole capstone is that the machine is trustworthy not because any one stage is perfect, but because each stage converts a hard distributed problem into a local invariant plus an observable contract. The order service commits a local transaction. Debezium reads a local log. Kafka Streams compute from a local state store. The WebSocket service fans out a local SharedFlow. Settlement reconciles against a local ledger. None of these stages solves the global problem alone; each owns its piece and hands a clean contract to the next. The Senior engineer’s mental model is the composition of those contracts, and the ability to reason about that composition, under pressure, across team boundaries, is what the title is awarded for.
A final reframe ties the technical thread to the mission. Each of those local invariants exists so that a customer who has never invested before can tap a button on a phone and, a second later, own a piece of a company, with every euro accounted for, every regulatory rule satisfied, and every system downstream eventually told. The complexity we walked through is in service of making that tap feel simple. A senior keeps that orientation when the design gets hard, because the reason the firm tolerates ten coordinated stages instead of one simple script is precisely that the tap must be both simple for the customer and safe under regulation. Lose sight of that and the architecture becomes complexity for its own sake. Hold it and every design choice has a reason the customer would recognize.
Mastery Questions
The cards below are not trivia. Each one asks you to reproduce a piece of the mental model a Senior engineer carries: trace the flow, defend a design choice, or diagnose a seam failure. Work through them by speaking the answer aloud, as you would in the system-design round, and only then check the back. If you cannot reconstruct the invariant, the contract, or the failure mode for a stage, return to that stage before moving on.
Question
Trace one trade end to end and name the invariant each stage owns.
Answer
The mobile client owns idempotency of the order intent. The order service owns regulatory fail-closed and the atomic ledger-plus-outbox transaction. The matching engine owns deterministic price-time priority as a pure function of an ordered input. The outbox owns durability of the event row inside the business transaction. Debezium owns low-latency capture from the WAL without extra query load. Kafka Streams own the derived state computation in isolation. The WebSocket service owns one Redis subscription fanned to many clients via SharedFlow. Settlement owns reconciliation against the custodian with breaks routed to review. Observability owns the cross-team trace that ties every hop together.
Sources & evidence9 claims · 7 cited
The end-to-end flow composes mechanisms backed by named Trade Republic sources (outbox, Debezium, YTM streaming, mobile-first, batch jobs); the synthesis narrative is internal-reasoning and carries no claim.
- Trade Republic ships native iOS in Swift and Android in Kotlin with Jetpack Compose, and the mobile client is a first-class mobile-first product surface rather than a thin shell over a web app.verified
- The outbox pattern writes the business row and an event row in one PostgreSQL transaction so both commit atomically, because Kafka cannot participate in a database transaction and a publish is a non-rollback side effect that would create phantom events or long-running transactions.verified
- Debezium streams the outbox table via PostgreSQL logical replication from the write-ahead log, capturing committed changes with low latency and near-zero extra table-query overhead compared with polling, and the outbox payload is stored as BYTEA so Debezium passes pre-serialized Avro bytes to Kafka without parsing, avoiding Debezium-level poison pills from schema mismatch.verified
- A Kafka Streams topology consumes raw quotes and trade events, computes derived state such as yield to maturity per ISIN, and sinks results to a Redis cluster using sharded pub and sub rather than plain pub and sub because plain pub and sub broadcasts every message to every node and hurts scalability at high fan-out.verified
- The real-time fan-out to clients uses Ktor WebSockets with a Kotlin SharedFlow per instrument, fed by exactly one Redis subscription per service instance that is created on first client subscription and torn down when the last client leaves, keeping resource cost independent of client count.verified
- Trade Republic runs settlement and reconciliation batch jobs as horizontally scaled, partitioned workers on Kubernetes for reliability and efficiency at scale, rather than as a monolithic ingestion service that cannot scale independently and must restart the whole batch on partial failure.verified
- Documented candidate and system failure modes at Trade Republic include not handling transaction rollbacks on failure, assuming all exchanges and custodians provide consistent data formats, building a monolithic ingestion service instead of a horizontally scalable one, and focusing on personal career growth over the democratize-finance business mission.verified
- Trade Republic operates an autonomy-plus-alignment model with about 280 engineers in domain teams, a Tech Radar standardizing technology choices, and an architecture forum of Senior and Senior-plus engineers reviewing significant decisions recorded as RFCs and ADRs, which is the structure under which a senior owns incidents across team boundaries.verified
- The end-to-end composition is trustworthy because each stage converts a hard distributed coordination problem into a local invariant plus an observable contract, and the senior mental model is the ability to reason about how those local contracts compose across team boundaries under pressure.internal reasoning
Cited sources
- State of Android at TR (2024 edition) and iOS engineering posts · Trade Republic Engineering
- Streaming Outbox Events from Postgres to Kafka with Debezium · Trade Republic Engineering (Andrei Sukharev)
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)
- Real-Time Bond Yield to Maturity: Streaming with Kafka and Redis · Trade Republic Engineering (Luis Jacintho)
- Scaling batch jobs for reliable and efficient processing · Trade Republic Engineering
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)
- From Architecture to Tech Radar: How Trade Republic uses Tech Radar and Architecture processes · Trade Republic Engineering (Daniela Valero)