On this path · System Design 1. Design a Real-Time Notification System
Design a Real-Time Notification System
Millions of users, price-alert plus trade-execution plus market-news fan-out, and latency and reliability tradeoffs.
Learning outcomes
The real Trade Republic prompt is to design a real-time notification system for millions of users that delivers price-alert, trade-execution, and market-news notifications with low latency, while considering scalability, reliability, and fault tolerance. Most candidates can sketch a fan-out queue. Fewer notice that the three notification classes have nothing in common on their latency, durability, and volume axes, and that a single uniform pipeline is the trap. This page teaches you to separate the classes first, then design each delivery path to its own guarantee.
After studying this page, you can:
- Classify the three notification classes by their latency tolerance, durability requirement, and fan-out shape, and defend a distinct delivery path for each.
- Estimate capacity for the combined load and show where each megabyte and each millisecond goes.
- Map the Trade Republic streaming machinery, Kafka, Redis Sharded Pub/Sub, and Ktor WebSockets, onto the in-app delivery path.
- Choose delivery guarantees per class and defend idempotency, deduplication, and backfill on reconnect.
- Reason about push via APNs and FCM versus in-app streaming, and where each belongs.
- Anticipate the failure modes a Staff engineer probes: lost pushes, duplicate trade fills, thundering-herd news bursts, and stale alert thresholds.
Before we dive in
A notification is a message a user opted to receive and expects to trust. That second clause is what makes the design hard. A price alert that arrives ten seconds late still earns its keep; a trade-execution confirmation that never arrives, or arrives twice, breaks the user’s trust in their broker. The prompt hands you three notification classes that look the same on the surface, a payload plus a destination, and are opposites underneath. Treating them identically is the first and most common failure.
The origin story of every notification system is the same. A triggering event happens somewhere in the platform: a quote crosses a threshold, a trade fills, a news desk publishes. Something must observe that event, decide who cares, render a message, and push it to a device that may be asleep, offline, or actively using the app. The engineering problem is that those four steps have different cost profiles, and the “who cares” step for a broadcast news event fans out to millions while the same step for a trade execution fans out to exactly one. A pipeline sized for the broadcast will waste resources on the unicast, and a pipeline sized for the unicast will collapse under the broadcast.
The senior move, which this page teaches, is to refuse the single pipeline. You design one intake and three delivery paths, each tuned to the guarantee its class needs. The intake is shared because triggers come from many sources and you want one place to record them. The delivery is split because the classes disagree about latency, durability, and volume, and a design that ignores that disagreement ships broken notifications.
Consider the concrete failure of a unified pipeline. A central-bank rate decision triggers a market-news burst to all one million connected clients. At the same moment, a thousand trades fill and their notifications are sitting in the same queue behind the news burst. The trade notifications arrive ten seconds late because the news burst consumed the delivery capacity. The unified pipeline’s latency for the trade class depends on the burst load from the news class, which is invisible to the trade team and unpredictable. A senior cannot defend a design where one class’s latency depends on another class’s volume, because the product guarantees for trade executions, must arrive within seconds, are violated by a news event that neither the trade team nor the matching engine controls. The separation of delivery paths is the only defense that makes the latency guarantee independent of the news volume.
The second failure of a unified pipeline is cost. Trade-grade durability, transactional outbox writes with Debezium CDC, is expensive per message. Applying it to every price alert, where the user tolerates loss, wastes infrastructure on a guarantee the product does not need. Applying it to every market-news broadcast wastes even more, because the broadcast volume is orders of magnitude above the trade volume. The unified pipeline either pays the trade-grade cost for every notification class, which multiplies the infrastructure budget, or it drops to a lower guarantee for all classes, which under-protects trade executions. Splitting the paths lets each class pay only for the guarantee it needs, which is the engineering discipline of matching cost to requirement.
Clarify the requirement before you draw a box
Before any architecture, separate the three classes and pin a number to each axis. This is the step that signals you are a senior, because a junior reaches for the whiteboard and a senior reaches for the requirements table first.
Price-alert notifications fire when a user-defined threshold on a quote stream is crossed. The user sets “tell me when Apple hits 200 dollars” and waits. The profile is high fan-out, because a popular stock may have hundreds of thousands of thresholds set on it, and a single quote update can fire many alerts. The latency tolerance is forgiving, a few seconds is acceptable, because the user is not acting on the alert in milliseconds; they are glancing at their phone. The durability requirement is weak; a missed price alert is an annoyance, not a financial loss, so at-most-once delivery is acceptable and redelivery on reconnect is a nicety, not a contract.
Trade-execution notifications fire when an order the user placed fills, partially or fully. The profile is per-user, exactly one recipient, and the guarantee is must-not-lose and must-not-duplicate. If the user sells a position and the confirmation is lost, they may double-sell; if the confirmation is duplicated, they may believe the system is broken. Latency must be low, a second or two, because the user is watching the order and anxiety rises fast. This class is where idempotency and durability live, and where the design earns its keep.
Market-news notifications fire when the editorial desk publishes a market-moving story or a corporate action lands. The profile is broadcast, potentially to every user, and the volume is high during volatile sessions. The latency tolerance is moderate, seconds to a minute. The durability requirement is mixed: the news itself is not critical, but the impression of a live product is, so a best-effort delivery with a durable fallback to a “missed notifications” inbox on reconnect is the right contract.
Three trigger sources and three delivery paths make the architecture.
The diagram is the load-bearing decision. Three triggers enter, three notifiers process them on their own terms, and two sinks, the in-app WebSocket and the push gateway, serve both. A single notifier trying to handle all three would either over-engineer the price alert or under-protect the trade execution.
The separation is not binary; it is a three-way split that the design table makes explicit. Price-alert notifications are latency-forgiving, durability-weak, and fan-out-heavy on popular instruments. Trade-execution notifications are latency-sensitive, durability-critical, and fan-out-per-user. Market-news notifications are latency-elastic, durability-soft, and fan-out-global. The class table is the reference a senior returns to when questioned about any design choice, because each choice, what delivery guarantee to offer, what storage engine to use, what retry policy to adopt, has a different answer for each class, and the class table is what forces the discipline to give the right answer for each.
A trap lurks when a single pipeline tries to serve all three classes.
With the classes separated, you can answer the non-functional requirements precisely. Scalability means the broadcast path scales to millions of concurrent WebSocket connections and the alert engine scales its threshold matching linearly with quote volume. Reliability means the trade-execution path never loses or duplicates a notification, even across service restarts. Fault tolerance means a Redis outage degrades the price-alert path to best-effort push without taking down trade executions, and a push-gateway outage queues in the outbox for later delivery.
Capacity estimation in round numbers
Pin round numbers to each class so the design has a budget to defend. Assume ten million users, with one million concurrently connected through the in-app WebSocket at peak. A volatile market day is the design point, because that is when all three classes spike together.
Price alerts: assume the average user sets five thresholds, so fifty million thresholds live in the system. A popular instrument updates its quote ten times per second during volatility, and a single update may match a few thousand thresholds after a sharp move. In the worst second, across the hot instruments, the alert engine fires on the order of one hundred thousand alerts. At a few hundred bytes each, that is tens of megabytes per second of alert payload, sustainable on a modest Redis cluster.
Trade executions: assume Trade Republic processes a few thousand trades per second at peak, a number consistent with a large European neobroker during a busy session. Each produces one notification, so a few thousand per second is the ceiling, and the volume is trivial compared to alerts. The constraint here is not throughput, it is correctness.
Market news: a burst event, such as a central-bank decision, fans out to all one million connected clients. One news item replicated one million times is one million messages in a few seconds. This is the design point for the broadcast path, and it is what forces Sharded Pub/Sub rather than plain Redis Pub/Sub.
WebSocket connection cost is the hidden number. One million open TCP-plus-TLS connections held by Ktor servers on Kubernetes pods consumes roughly a hundred kilobytes of state per connection, so about a hundred gigabytes of memory across the fleet just to hold the sockets. That number, more than the message volume, decides how many pods you run and why a single Redis connection per instance matters: you cannot afford one Redis connection per client.
The push path has different cost drivers. APNs and FCM each enforce rate limits per device token and per connection. For a burst of one million push notifications, the gateway must fan across multiple provider connections and respect backoff and retry. The push gateway is sized for the burst rate, not the sustained rate, because the burst is the design constraint and the sustained notification rate is far lower. Typical APNs limits allow a few hundred notifications per second per connection, so a burst of a hundred thousand requires hundreds of concurrent connections to the provider. The gateway must manage those connections with TLS renegotiation, token invalidation, and error-driven backoff, which makes the push path operationally heavier than the WebSocket path per notification delivered.
The WAL retention budget is another hidden number that a senior does not miss. Trade-execution notifications that flow through Debezium from the outbox to Kafka consume Postgres WAL space while the connector is catching up. During a Kafka outage, the WAL grows at the rate of the outbox insert rate, which is the trade rate. A one-hour Kafka outage at a few thousand trades per second means gigabytes of WAL growth, which must fit in the provisioned WAL disk or the database stalls. Naming this budget shows you understand that the notification system’s durability guarantee rests on the database’s disk budget, not just on a design pattern.
High-level architecture
The architecture has five layers, each with a job that the next layer depends on. Layering is the senior move because it lets you argue about each guarantee in isolation and lets each layer fail independently.
The trigger layer is where events originate: the quote stream from the market-data pipeline, trade-fill events from the matching engine, and news plus corporate-action events from editorial and reference-data services. None of these knows anything about notifications; they just publish. This separation matters because the notification system must not couple its health to the quote stream’s health, and a clean Kafka topic boundary lets each side evolve.
The classification layer consumes triggers and routes them to the right notifier. For price alerts it runs the threshold match; for trade fills it fans to the single owning user; for news it expands to the subscriber set. This is where the three classes diverge into their own logic, and where a shared library of notification policies, dedup keys, and rate limits lives so the notifiers stay consistent.
The delivery layer is split. The in-app path streams through Redis Sharded Pub/Sub into Ktor WebSockets, one connection per instance fanning to many clients via Kotlin SharedFlow, exactly the pattern Trade Republic uses for yield-to-maturity streaming. The push path sends through APNs for iOS and FCM for Android, for users whose app is backgrounded or offline. A single notification often travels both paths: the WebSocket for the live view, and the push to wake the device.
The durability layer holds what must survive. Trade-execution notifications are written to an outbox table in the same transaction as the trade, so Debezium can stream them to Kafka with the database transaction guarantee, and a missed delivery is retried from the log. Price alerts and news use a short best-effort queue, because their value decays and durability is not worth the cost.
The client layer reconciles. On reconnect, the client asks for notifications it missed since its last acknowledged sequence number, and the server replays from the durable store for trade executions and from a short retention window for the rest. This backfill is what turns a lost WebSocket into a correct experience rather than a silent gap.
The separation into five independent layers is what makes the system testable and resilient. Each layer owns its guarantee: the trigger layer owns event ingestion, the classification layer owns routing, the delivery layer owns throughput, the durability layer owns persistence, and the client layer owns reconciliation. A failure in any one layer does not cascade into the others, because each communicates through a bounded queue with a different durability and retention policy.
Deep dive: the trigger fan-in and the delivery fan-out
The hard subsystem is the join between trigger fan-in and delivery fan-out, because that is where the three classes’ opposing requirements collide. Walk each path on a concrete case.
Price-alert matching under a quote burst
The naive design evaluates every threshold against every quote. With fifty million thresholds and ten quote updates per second on a hot instrument, that is half a billion evaluations per second, which is absurd. The senior design indexes thresholds by instrument and by price bucket, so a quote update for one instrument only touches the thresholds set on that instrument, and only those whose bucket the new price has entered.
The matching engine holds, per instrument, a sorted structure of thresholds keyed by price. When a quote arrives, it walks the range of thresholds the quote has crossed since the last quote, fires each once, and clears or arms it according to the user’s repeat policy. The work per quote is proportional to the number of thresholds actually crossed, not the total set, which is what makes the system tractable. Each fired alert becomes a small payload on a Redis channel keyed by user, and the WebSocket path delivers it.
The rate-limit subtlety: a fast-moving stock can cross a threshold, retreat, and cross again within a second. Firing on every cross spams the user. The alert policy debounces by arming a cooldown per threshold, so a re-cross within the cooldown is suppressed. This is a product decision surfaced as an engineering parameter, and naming it shows you understand that notification systems are policy engines, not just pipes.
The threshold data structure is a concrete design decision that the interviewer may probe. One approach is a sorted set per instrument in Redis, where each member is a threshold with a score equal to the threshold price. When a quote arrives, the engine issues ZRANGEBYSCORE to find all thresholds the new price has crossed since the last quote. This is simple and fast for a single check, but it requires reading and writing the sorted set on every quote, which becomes the bottleneck for a hot instrument with hundreds of thousands of thresholds. The alternative is an in-memory structure in the alert engine: a per-instrument balanced tree of thresholds, where the engine walks the range the quote crossed and fires alerts in batch. The in-memory tree avoids the Redis round trip for every quote, at the cost of rebuilding from the database on restart. The production choice depends on the volume: for a broker with tens of millions of thresholds, the in-memory tree is the proven choice because it eliminates the network hop from the critical path.
Regardless of which data structure the engine uses for matching, the fired alerts land on Redis channels, and the matching engine itself is not involved in delivery. The matching engine evaluates and publishes to the channel; the WebSocket path handles delivery. This separation keeps the matching engine fast and the delivery path decoupled, which is consistent with the separation pattern across the rest of the Trade Republic platform.
Trade-execution delivery with a must-not-lose guarantee
A trade fills in the matching engine. The engine, following the consistency pattern in Consistency, Transactions, and Rollbacks, writes the fill to its database and writes a notification outbox row in the same transaction. Because the outbox row commits with the trade, the notification’s existence is guaranteed by the same database atomicity that guarantees the trade. This is the outbox pattern documented in The Outbox Pattern, Deeply, applied to notifications.
Debezium streams the outbox row to a Kafka fill-notification topic. A notifier consumer reads the topic, renders the notification, and delivers it through both the WebSocket, for the live view, and the push gateway, for the offline case. The delivery is idempotent because the outbox row carries the trade’s unique id as its dedup key, so a redelivery after a consumer restart does not produce a duplicate notification on the client. The client acknowledges by sequence number, and unacknowledged notifications are retried from the Kafka topic, which retains them for days.
The hazard this design avoids is the dual-write. If the notifier wrote to the WebSocket and then separately marked the notification sent, a crash between the two would either lose the notification or deliver it twice. The outbox collapses the two into one transactional fact, the row exists, and delivery becomes a retry-safe projection of that fact.
Market-news broadcast through Sharded Pub/Sub
A news item publishes to the broadcaster, which must reach all one million connected clients. Plain Redis Pub/Sub broadcasts every message to every Redis node, which is why Trade Republic chose Sharded Pub/Sub for yield streaming: the shard ownership of a channel determines which node handles it, so the load spreads across the cluster instead of replicating to every node. The same choice holds for news.
Each Ktor instance subscribes, through a single Redis connection, to the news channel. When a news message lands, the instance fans it to every locally-connected client via a Kotlin SharedFlow, one logical stream per news topic that many collectors share. The single-connection-per-instance rule is what keeps Redis connection count equal to the pod count, not the client count, which is the difference between thousands of connections and millions. This pattern is documented in the streaming architecture; see Redis Cluster and Real-Time Fan-Out.
The number of Redis subscriptions equals the number of pods times the number of active channels, never the number of clients. Inside each pod, a SharedFlow fans one channel’s updates to thousands of connected WebSockets. This is the design choice that makes million-client fan-out affordable, and it is the detail a Staff engineer will probe.
Push versus in-app
In-app streaming is for the user who has the app open; push is for the user who does not. A trade-execution notification must reach the user regardless, so it always goes to push and, if the app is open, also to the WebSocket. A price alert goes to push only if the user has not opened the app recently, to avoid double-notify fatigue, and to the WebSocket if they have. Market news goes to the WebSocket for engaged users and to push only for items the user has explicitly opted into, because pushing every news item to every user would burn the push budget and get the app uninstalled.
The push gateway is a separate service from the streaming path because APNs and FCM have their own rate limits, retry semantics, and failure modes, including token invalidation when the user uninstalls the app. A dedicated gateway handles token refresh, batches where the providers allow it, and reports permanent failures back so the system can stop sending to dead tokens.
Push gateway architecture and token lifecycle
The push gateway maintains a pool of provider connections, each authenticated with a provider-specific credential. For APNs, one connection per app bundle id using HTTP/2 multiplexing, which pushes as many requests as needed per connection without the per-connection limits of the old binary protocol. For FCM, connections use HTTP/1.1 with keep-alive and batch where the API allows. The gateway is stateless with respect to the notification payload: it receives a notification target from the notifier, looks up the device token and provider from the device table, and pushes.
Token invalidation is the operational concern that makes push hard. When a user uninstalls the app, the next push to their token returns a permanent failure from the provider. The gateway must record this and stop sending to that token, or it wastes resources and inflates the provider’s latency percentiles with dead connections. The token table is updated on every successful app launch, when the device registers a new token, and on every permanent failure, when the gateway marks the token as invalid. A periodic sweep removes stale tokens that have accumulated permanent failures, keeping the token table lean.
Rate limiting is per-provider and per-token. APNs enforces a rate limit per connection and may throttle a sender that exceeds it. The gateway tracks its send rate per connection and backs off when approaching the limit, queuing notifications internally rather than sending and being throttled. The queue is bounded and spillage is logged, because a queue that grows unboundedly behind a throttled connection consumes memory and delays newer notifications. A senior designs the queue with a TTL so a notification that has waited too long for that connection is either discarded or routed to a fallback channel.
The cold analytics path for notification delivery
Not every notification needs to be delivered in real time. Trade-execution notifications are on the hot path because the user is waiting. Price-alert notifications that the user did not see while the app was closed, and market-news notifications the user missed during a burst, need a cold path that aggregates them into a digest or an inbox the user can open later.
The cold path is a separate consumer of the fill-notification Kafka topic and the alert stream that writes to a per-user notification inbox in Postgres. The inbox stores every notification the user was supposed to receive, with a flag for whether it was delivered via the hot path. When the user opens the app after being away, the client requests the inbox content since the last check, and the server returns the undelivered notifications in sequence order. The inbox is the durable fallback for the in-app streaming path, and it is what makes the system complete: the hot path provides low latency, and the cold path provides completeness.
The cold path’s storage must be partitioned, because a single inbox table at Trade Republic’s scale would become a write bottleneck. Partition the inbox by user id hash, with each partition storing its slice of notifications. The retention policy is set by the notification class: trade-execution notifications are retained indefinitely for regulatory reasons, while price-alert and market-news notifications are retained for a configurable window, typically a few days, after which they are archived or deleted. The cold path is allowed to be minutes behind the hot path, because the user does not open the inbox immediately; they open it when they re-engage with the app, and completeness matters more than latency there.
Combining push and in-app for the same notification
A single trade-execution notification typically travels both paths: the WebSocket delivers it immediately to the live client, and the push arrives moments later as a redundant wake-up. The client collapses the duplicate on the dedup key so the user does not see two notifications. The redundancy is deliberate: if the WebSocket dropped before delivery, the push is the safety net. If the WebSocket delivered, the push is silently collapsed. This dual-delivery pattern is the standard mobile notification architecture for sensitive events, and articulating it shows you understand that reliability in this domain means over-informing rather than under-informing.
Data model
The data model has four entities, one per concern, and keeping them separate is what lets each path be reasoned about alone.
The notification row’s dedup key is the load-bearing field. For a trade fill it is the trade id; for a price alert it is the threshold id plus the cooldown window id; for news it is the news id plus the user id. The client uses the dedup key to collapse redeliveries, and the server uses it to make delivery idempotent. The sequence number is per-user and monotonic, assigned at write time, so the client can request “everything after sequence N” on reconnect and the server can serve it from the durable store.
The sequence number assignment is a subtle design choice with operational consequences. A global sequence would require a coordination service and would be a bottleneck. A per-user sequence avoids coordination, because each user’s notifications are independent. The sequence is assigned by the outbox insert time for trade-execution notifications and by the alert engine eval time for price-alert notifications. The assignment must be monotonic within a user timeline, which means the writer must serialize writes per user. For trade-execution notifications, serialization is natural because trades for a user are serialized by the ledger. For price-alert notifications, the alert engine serializes per user to avoid sequence gaps, which caps per-user alert throughput but is acceptable because a single user rarely receives more than a handful of alerts per second.
The device table has a subtlety that matters at scale. A user may have multiple devices: a phone and a tablet, or a replaced phone with an old token. The push gateway fan-out sends the notification to every active device token for the user. The trade-execution notification is idempotent per device because it carries the same notification id on every push. The WebSocket path, by contrast, delivers to whichever device has the open connection, because only one WebSocket is typically active per user at a time. The design choice is to let the push path cover all devices redundantly and let the client collate, rather than routing per device at the server. The cost is extra push notifications; the benefit is server-side simplicity and the guarantee that the user sees the notification regardless of which device they are using.
Tradeoffs and alternatives
A single unified notification pipeline, one queue and one worker pool for all classes, is the alternative a junior proposes. Its appeal is simplicity. Its failure is that the trade-execution path inherits the latency and durability of the busiest class, so a news burst starves trade confirmations, or the trade path’s heavy transactional guarantees are paid on every price alert, wasting resources. Splitting the paths costs engineering complexity and buys isolation, and for a broker the isolation is worth more than the simplicity.
Plain Redis Pub/Sub instead of Sharded Pub/Sub is simpler to operate but does not scale: every node receives every message, so adding nodes to handle more clients increases per-node message load rather than spreading it. Sharded Pub/Sub adds the complexity of channel-to-shard routing but is the only way to scale broadcast to millions. The trade is operational maturity for capacity, and at Trade Republic’s scale the capacity wins.
A polling client instead of a WebSocket is simpler to build and works through corporate proxies, but it imposes a latency floor equal to the poll interval and a load floor equal to the poll rate times the client count. WebSockets invert the trade: continuous connection cost in exchange for push latency. For a real-time product, the push latency is the product, so the WebSocket is correct, with polling as a degraded fallback.
Writing trade notifications directly to Kafka without the outbox is the tempting shortcut. It fails because the trade write to Postgres and the Kafka write cannot share a transaction, so a crash between them produces a trade with no notification or a notification with no trade. The outbox is the cost of correctness; skipping it is the documented dual-write failure.
A Ktor-based notification service versus a dedicated push notification service such as OneSignal or Firebase is the build-versus-buy tradeoff. A third-party service handles device token management, rate limiting, and provider connections out of the box, which reduces operational burden. Trade Republic’s documented engineering philosophy, however, favors owning the core delivery path because notification reliability is a product differentiator. A third-party service introduces a dependency that can fail or change its pricing, and debugging a missed trade notification across a third-party boundary is harder than debugging it inside the platform. The build choice costs engineering time and operational maturity; it buys full control of the delivery path and the ability to tune it per notification class.
A client library that communicates over a custom binary protocol instead of WebSocket is theoretically more efficient but is a non-starter for a consumer app where the client runs on iOS and Android and the network path goes through NATs, proxies, and carrier-grade middleboxes. WebSocket is universally supported, well-tested through NAT, and has mature library support on both platforms. A custom protocol would require a custom keep-alive, reconnection, and backpressure implementation on both client and server, which is a large investment for marginal benefit. The senior choice is WebSocket with standard extensions, not a custom protocol.
A push-only system without the in-app WebSocket path is simpler to build: no persistent connections, no SharedFlow, no Redis Pub/Sub. The trade is that push latency is unpredictable because it depends on the provider’s delivery timing, and the user cannot see the notification until the system tray displays it. For a product that is real-time by definition, the push-only path would deliver notifications seconds to minutes late, which is unacceptable for trade executions where the user is watching. The in-app WebSocket path provides sub-second delivery for the engaged user, and the push path covers the backgrounded user. The two paths together give the user a consistent real-time experience regardless of whether the app is foregrounded.
Failure modes and what breaks at scale
A Redis cluster outage takes down the in-app streaming path for price alerts and market news. The graceful degradation is to fall back to push for alerts and news, accepting higher latency, while the trade-execution path continues on its Kafka outbox unaffected because it never depended on Redis. Designing so that one outage degrades rather than collapses is the mark of a senior design, and the class separation is what makes it possible.
A push-gateway outage, where APNs or FCM is unreachable, must not lose trade notifications. The Kafka fill-notification topic retains messages, and the notifier retries until the gateway recovers. The risk is that a delayed trade confirmation looks to the user like a failed trade, so the in-app path must show the fill immediately for connected users, and the push is the wake-up channel, not the source of truth.
A thundering herd on a news burst can saturate a single Redis shard if too many channels hash to it. The mitigation is channel-key design: hash news topics so they spread, and monitor shard hotness. If a single topic is genuinely the whole audience, shard by subscriber-group within the topic, accepting that the broadcaster must fan to each group. This is a real scaling limit and naming it shows depth.
Stale alert thresholds cause phantom alerts when a stock splits or a corporate action changes its price scale. The alert engine must consume corporate-action events and adjust or invalidate thresholds, or a one-for-ten split fires every threshold below the new price. Coupling the alert engine to the reference-data feed is not optional; it is a correctness requirement.
Duplicate trade notifications from consumer restarts are prevented by the dedup key, but only if the client respects it. A client that renders every incoming message without checking dedup will show duplicates. The contract is bilateral: the server is idempotent, and the client must collapse on the same key.
A Debezium connector outage that stalls the outbox publisher for the trade-notification path is a systemic failure mode. The outbox table grows unboundedly, and the unpublished index, even if correctly built on a partial index of published_at IS NULL, eventually grows large enough to bloat. The senior defense is to monitor the outbox depth per connector and to alert when it exceeds a threshold, because the notification system’s correctness depends on the outbox being drained, and an undrained outbox is a latency failure that looks to the user like a missing trade confirmation. The outbox design from the Trade Republic engineering blog, which partitions the outbox by published_at and uses bigint for the id, is the production hardening that this failure mode requires.
Notification fatigue from excessive price alerts is a product failure that manifests as an engineering problem. If the alert matcher fires too many alerts per user per day, the user disables notifications or uninstalls the app. The system must enforce per-user and per-instrument rate limits at the classification layer, debouncing fast re-crossings as described above and capping the daily alert count per user. This is a rare case where the engineering design directly determines product experience, and a senior names the coupling rather than treating it as someone else’s problem.
Sequence number overflow on the per-user sequence is a failure mode at extreme longevity. If the sequence is a 32-bit integer and a single user receives millions of notifications over their lifetime, the sequence wraps around and the client’s “after this sequence” request becomes ambiguous. The mitigation is to use a 64-bit sequence, which is essentially inexhaustible for any realistic user lifetime, and to make the sequence database column a bigint. The Trade Republic outbox design explicitly calls out the int4-to-int8 migration for this reason, and the same caution applies to the notification sequence.
Dedup key collision across notification classes is a subtle bug. If the dedup key for a trade fill and the dedup key for a price alert happen to collide, because both use the same field as the key and the field values overlap, the client collapses one notification thinking it is a duplicate of the other. The prevention is to namespace the dedup key by class: a trade fill’s dedup key is fill:{trade_id}, a price alert’s is alert:{threshold_id}:{cooldown_id}, and a news item’s is news:{news_id}. The colon prefix ensures that keys from different classes never collide, and the convention is enforced at the server and verified by integration tests.
A provider-side outage of APNs or FCM is outside the system’s control but must be handled gracefully. The push gateway retries with exponential backoff and jitter, and after a configurable number of consecutive failures, it marks the notification as undelivered and logs the incident for operational review. The trade-execution notification is not dropped; it remains in the Kafka topic and is retried when the provider recovers. The user who is in-app sees the notification on the WebSocket path, so the provider outage is invisible to the connected user and only affects users who are backgrounded. This layered degradation, where each path fails independently and the user’s experience degrades only as much as necessary, is the senior design property.
Alert engine cold start and recovery
The in-memory threshold tree must be rebuilt when the alert engine restarts. Loading fifty million thresholds from the database on startup can take minutes, during which no price alerts fire. The mitigation is to load thresholds lazily by instrument: the engine loads thresholds for an instrument on the first quote arrival for that instrument, so the cold start penalty is paid per instrument rather than all at once. A popular instrument’s thresholds are loaded within seconds of startup; a long-tail instrument with no quotes in the first hour incurs no startup cost. The lazy loading policy is a judgment call: it delays the first alert for a cold-start instrument by the threshold load time, but it avoids a minutes-long startup pause. A senior names this trade-off rather than pretending the system starts instantly.
The threshold database must be partitioned for the same reason the dedup table is partitioned. A single table of fifty million thresholds with frequent reads by instrument and user would become a performance problem. Partitioning by instrument id hash distributes the read load across the database, and the lazy loader reads only the partition it needs. The partitioning strategy follows the same hash-vs-list analysis documented in the Trade Republic engineering blog, where hash partitioning on a natural key, here the instrument id, wins because it distributes evenly and preserves primary keys.
Data retention and privacy
Notification data carries personal information, the user’s device token, their alert preferences, and the record of trade fills they were notified about. Regulatory requirements may mandate retention of trade-execution notification records for years, while price-alert preferences can be deleted when the user closes their account. The system must support per-class retention policies and a data deletion workflow that purges all notification data for a user on account closure.
The device token table is the privacy-sensitive component. A token identifies a specific device and can be linked to the user. When the user deletes their account, the token table must be purged immediately, not just marked invalid, because a retained token is a privacy liability. The purge is a database deletion that cascades to the push gateway’s token cache, ensuring no future notification attempts for a deleted user. This is not a feature; it is a regulatory mandate that a senior designs for from the start rather than retrofitting.
Operational concerns
Observability is per-class. Track end-to-end latency separately for alerts, fills, and news, because an aggregate p99 hides a broken fill path behind a healthy alert path. Track delivery success per channel, WebSocket versus push, and track the dedup collision rate, which should be near zero; a rising rate signals a dedup-key design bug.
Backpressure on the alert matcher, when quote volume outpaces evaluation, must shed load gracefully rather than queue unboundedly. The policy is to drop the oldest unprocessed quote for an instrument and evaluate only the latest, because a price alert on a stale quote is worse than a delayed alert on the current price. This is a domain-specific backpressure rule, and articulating it shows you understand that backpressure policy is a product decision.
Capacity planning keys off the WebSocket connection count, which sets the pod fleet, and the burst fan-out, which sets the Redis cluster size. The trade path’s capacity is trivial by comparison and should not drive sizing. A common error is to size the whole system off the trade-execution volume and then be surprised when a news burst overwhelms it.
Cost allocation is an operational concern that surfaces when the system is shared across teams. The notification pipeline serves multiple internal customers: the trade desk owns fill notifications, the editorial desk owns market news, and the product team owns price alerts. Shared infrastructure, the Redis cluster, the WebSocket fleet, the push gateway, is paid from a central budget, but each team’s usage pattern drives different cost profiles. A senior anticipates the question of chargeback or fair-share allocation and designs the observability layer to attribute cost per notification class, so each team can see and optimize its own consumption. This is the type of organizational concern that a Staff engineer probes because it determines whether the system is sustainable as the company scales, not just whether it works.
Deployment safety means the WebSocket path must support rolling reconnects without dropping notifications, which is why the backfill-on-reconnect contract exists. A pod going down should look to the user like a brief blip followed by a catch-up burst, not a gap. The Ktor service is designed for the Kubernetes lifecycle: it handles SIGTERM by draining existing WebSocket connections, refusing new ones, and waiting for the in-flight SharedFlow buffers to flush before exiting. The client library implements a reconnect with exponential backoff and jitter, so a fleet rollout does not produce a thundering herd of simultaneous reconnections.
Testing the notification system requires failure injection at every boundary. The team simulates a Redis cluster outage and verifies that trade notifications continue via the Kafka outbox path and that price alerts degrade to push gracefully. They simulate a push-gateway failure and verify that the Kafka fill-notification topic retains the events for retry. They simulate a Debezium connector stall and verify that the outbox depth alert fires before the WAL fills. These tests are not one-time exercises; they are part of the release pipeline, because every change to the notification pipeline risks breaking one of the three paths. A senior insists on automated chaos experiments for the notification system, because the three paths interact in ways that a unit test cannot catch.
Defending the design to a Staff engineer
A Staff engineer will probe three things, and you should preempt each. First, why three paths instead of one? The defense is the class separation table: the three classes disagree on latency, durability, and volume, and a unified path either over-engineers the cheap classes or under-protects the expensive one. The cost is three code paths; the benefit is isolation, so a news burst cannot starve a trade confirmation.
Second, how do you guarantee a trade notification is never lost or duplicated? The defense is the outbox: the notification row commits in the same Postgres transaction as the trade, Debezium streams it to Kafka, the notifier delivers idempotently on the trade-id dedup key, and the client acknowledges by sequence number with retry from the retained topic. Every step is either transactional or idempotent, so the only failure modes are retry-safe.
Third, how do you scale the broadcast to a million clients without melting Redis? The defense is Sharded Pub/Sub plus the single-connection-per-instance rule plus SharedFlow fan-out inside each pod. The Redis connection count equals the pod count, not the client count, and the shard routing spreads the channel load across the cluster. State the memory budget, roughly a hundred gigabytes to hold one million WebSocket connections across the fleet, to show you have done the arithmetic.
A fourth probe that separates seniors from strong mids is about idempotency boundaries. The Staff engineer asks: where does idempotency live, and what happens if the client and server disagree? The defense names three distinct idempotency scopes. The trade notification itself is idempotent on the trade id at the messaging layer, so the notifier can retry without fear. The push notification is idempotent on a notification id at the push gateway, so a retry to APNs or FCM does not double-deliver to the device. The in-app notification is idempotent on the dedup key at the client, so duplicate WebSocket messages are collapsed. These three scopes are independent and each has its own failure mode. The server and client disagree when the dedup key is not correctly generated or when the client clears its dedup cache on reinstall; the recovery is that the server replays from the durable store and the client treats the replay as authoritative, accepting that some dedup collisions may occur during the replay window.
The fifth probe is about cross-region disaster recovery. If the primary AWS region fails, how does the notification system recover? The honest answer is that the notification system is regional, not global. Trade-execution notifications are regionally sourced from the regional ledger and outbox, so a region failover requires the ledger and matching engine to failover first. Price-alert state, the fifty million thresholds, must be replicated to a stand-by Redis cluster in the secondary region, and the alert engine must run there with a warm copy of the threshold set. Market news is the easiest to fail because it is stateless broadcast. A senior states the recovery time objective per class: trade-execution notifications are bound by the ledger failover, price-alert notifications by the Redis replication lag, and market-news notifications by the time to activate the WebSocket fleet.
State it plainly: the system is one intake and three delivery paths, because the three notification classes share a trigger source but share nothing on their latency, durability, and volume axes, and the senior design refuses to let one class’s failure mode become another’s.
Common mistakes candidates make
- Treating all notifications as one queue. A single pipeline lets a news burst starve trade confirmations, or forces trade-grade durability onto every price alert. Class separation is the first and most important decision.
- Dual-writing the trade notification to Kafka outside the trade transaction. A crash between the trade commit and the Kafka write loses the notification or the trade. Use the outbox so the notification commits with the trade.
- Using plain Redis Pub/Sub for broadcast. It replicates every message to every node and does not scale past a few nodes. Sharded Pub/Sub is the production choice for million-client fan-out.
- Opening one Redis connection per WebSocket client. That is millions of connections and is the scaling cliff. One connection per pod, fanning through SharedFlow, is the correct shape.
- Forgetting backfill on reconnect. A dropped WebSocket looks like a silent gap to the user. The client must request missed notifications by sequence number and the server must replay from the durable store.
- Ignoring corporate actions on alert thresholds. A stock split rescales the price and fires every threshold below it. The alert engine must consume corporate-action events and adjust thresholds.
- Skipping the dedup key on the client. The server is idempotent, but if the client renders every message without collapsing on the dedup key, the user sees duplicates. Idempotency is a bilateral contract.
- Pushing every news item to every user. That burns the APNs and FCM rate budget and drives uninstalls. Push news only to users who opted in; stream it to the rest.
- Using a 32-bit sequence number for the per-user notification sequence. At extreme longevity, the sequence wraps around and the client’s after-this-sequence request becomes ambiguous. Use a 64-bit sequence.
- Not namespacing dedup keys by notification class. A trade fill and a price alert with the same key value are collapsed into one notification on the client. Prefix the dedup key with the class name.
- Assuming the push gateway can be stateless with respect to token lifecycle. Dead tokens must be recorded and suppressed, or the gateway wastes connections and inflates latency on permanent failures.
What the interviewer asks next
Expect to be pushed on ordering. The question: across a reconnect, how do you ensure the user sees trade notifications in the order they happened, not the order they were delivered? The answer is the per-user monotonic sequence number assigned at outbox write time; the client renders by sequence and the server replays in sequence from the durable store.
A second follow-up: what is the failure mode if Debezium falls behind by minutes? The trade-execution notifications are delayed, but because they are retained in Kafka and the client backfills on reconnect, no notification is lost; the user sees a catch-up burst. The price-alert and news paths, which do not depend on Debezium, continue. The design degrades on latency for one class without losing data for any.
A third, deeper probe: how do you avoid duplicate delivery when the notifier crashes after sending to the WebSocket but before checkpointing the Kafka offset? The answer is that delivery is idempotent on the trade-id dedup key, so the redelivered message is collapsed by the client, and the Kafka offset is checkpointed only after a successful send, so at-least-once with idempotent collapse is the guarantee. Exactly-once is a myth here; idempotent at-least-once is the engineering reality.
A fourth probe, about the push gateway: what happens when APNs or FCM changes its API or deprecates a protocol version? The answer is that the push gateway is versioned internally and connects to the provider through an abstraction layer that isolates the provider protocol. When the provider changes, only the adapter changes, not the notification pipeline. The older adapter is kept until the provider sunsets it, and the migration is a gateway deploy, not a pipeline redesign. A senior names the adapter pattern explicitly, because it shows awareness that external dependencies change and the design must absorb that change.
A fifth, behavioral probe: a product manager asks why a market-news notification that takes fifteen seconds to reach the user is too slow. The engineering answer is that the broadcast path through Sharded Pub/Sub and WebSocket fan-out is sub-second; the fifteen-second latency is either a push-gateway delay when the user is backgrounded, which is outside the system’s control and depends on the provider’s delivery timing, or a back-pressure delay when the news burst exceeds the Redis cluster’s capacity, which requires a capacity review and possibly a larger cluster. A senior does not defend a bad number; they diagnose which stage caused it and escalate to the team that can fix it.
A sixth probe, about personal data: a regulator asks what notification data is retained, for how long, and how it is deleted on account closure. The senior answer names the three data categories: device tokens, retained while the user has a registered device and purged on account closure; notification history for trade executions, retained per regulatory requirements and archived to cold storage after the retention period; and price-alert preferences, retained while the user has the product and deleted on account closure. The answer names the purge mechanism, a cascading database deletion that triggers token cache invalidation in the push gateway, and the retention enforcement, a scheduled cleanup that archives or deletes records past their policy window. Naming this anticipation shows the senior has thought beyond the engineering into the regulatory frame the system operates in.
A seventh probe, about testability: how do you verify the notification system works correctly without pushing real notifications to millions of users? The answer names three testing strategies. Integration tests verify each delivery path in isolation by subscribing a test WebSocket client and a test push client to the notifier and confirming the correct notifications arrive with the right payload and dedup key. Chaos experiments inject a Redis cluster failure and confirm the trade path continues through the outbox and the alert path degrades to push. Canary deployments push notifications to a small percentage of production users and monitor delivery success rates and latency before rolling to the full fleet. The combination of isolation tests, chaos experiments, and canaries gives confidence that the system works under load without risking the user experience.
Mastery Questions
Question
Why does the notification system use three delivery paths instead of one unified pipeline?
Answer
The three classes, price-alert, trade-execution, and market-news, share a trigger source but disagree on latency tolerance, durability requirement, and fan-out volume. A unified pipeline either over-engineers the cheap classes with trade-grade durability or lets a news burst starve a trade confirmation. Splitting the paths buys isolation so one class's failure mode cannot become another's, at the cost of three code paths.
Sources & evidence5 claims · 3 cited
Design based on real Trade Republic interview prompts and engineering blog sources (src_tr_interview, src_tr_ytm, src_tr_outbox1); design reasoning beyond sources is internal-reasoning and carries no claim.
- Trade Republic's real-time notification system design separates three notification classes (price-alert, trade-execution, market-news) into distinct delivery paths because they share a trigger source but disagree on latency tolerance, durability requirement, and fan-out volume.verified
- The in-app notification delivery path uses Redis Sharded Pub/Sub with one connection per Ktor pod, fanning updates to thousands of WebSocket clients per pod via Kotlin SharedFlow, following the same pattern Trade Republic uses for yield-to-maturity streaming.verified
- Trade-execution notifications are written to an outbox table in the same PostgreSQL transaction as the trade, so Debezium can stream them to Kafka with the database's atomicity guarantee, preventing dual-write loss.verified
- A single unified notification pipeline is the documented trap: a news burst starves trade confirmations, or trade-grade durability is over-applied to every price alert. Class separation is the first and most important design decision.verified
- One Redis connection per Ktor pod, not per client, caps the Redis subscription count at the pod count times the channel count, which is the design choice that makes million-client notification fan-out affordable.verified
Cited sources
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)
- Real-Time Bond Yield to Maturity: Streaming with Kafka and Redis · Trade Republic Engineering (Luis Jacintho)
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)