On this path · System Design 3. Design Real-Time Market Data Streaming to Clients
Design Real-Time Market Data Streaming to Clients
Kafka Streams to Redis to WebSocket pipeline, generalized from the yield-to-maturity architecture.
Learning outcomes
The task is to design the full real-time market-data streaming pipeline that delivers quotes, computed analytics such as yield to maturity, and live charts to millions of Trade Republic clients. The architecture to generalize is the one Trade Republic published for streaming bond yield to maturity: raw quotes into Kafka, a Kafka Streams topology that computes per-instrument analytics, a Redis cluster with Sharded Pub/Sub, and Ktor WebSockets that fan updates to clients through a single connection per instance backed by Kotlin SharedFlow. This page teaches you to scale that architecture from one analytic to the whole market-data surface and defend each stage’s cost and guarantee.
After studying this page, you can:
- Separate the streaming stages, ingestion, compute, fan-out, and delivery, and justify why each is a distinct tier.
- Defend Kafka Streams for the compute tier, including stateful aggregation and the tradeoffs versus a stateless consumer.
- Defend Redis Sharded Pub/Sub over plain Pub/Sub, and explain the single-connection-per-instance rule that caps Redis connections at the pod count.
- Explain on-demand subscription, where a Redis subscription is created on first client interest and torn down on last client departure.
- Reason about backpressure, late and out-of-order quotes, and recovery on client reconnect.
- Anticipate the Staff-engineer probes on hot-shard imbalance, historical chart backfill, and cost at million-client scale.
Before we dive in
A market-data stream is the product. When a customer opens a stock or bond detail page, they expect the price to move, the yield to update, and the chart to extend, all in real time. A stale quote is not a minor bug; it is a broken product, because the customer’s decision to buy or sell rests on the number they see. The engineering problem is that the raw feed from exchanges and vendors is enormous, mostly irrelevant to any single client, and frequently noisy, while the client wants a clean, computed, per-instrument view pushed the instant it changes.
The origin of the published Trade Republic architecture is exactly this gap. Bond customers do not care about the raw price quote; they care about yield to maturity, a number computed from the price, the coupon, and the time to maturity. Computing YTM in real time from the incoming quote stream, and pushing only the result to clients, is what makes the product usable. The published pipeline does this with Kafka Streams consuming raw quotes, computing per-ISIN YTM, and sinking the result to both Redis, for live streaming, and a Kafka topic, for the historical chart. That architecture generalizes to every computed market-data analytic, and the design task is to scale it to millions of clients and thousands of instruments.
The senior insight is that the pipeline is a series of funnels, each of which reduces volume. The raw quote feed is the widest funnel: thousands of instruments updating many times per second. The compute tier narrows it by collapsing many quotes into one analytic per instrument. The fan-out tier narrows it further by delivering only the instruments each client watches, not the whole market. The delivery tier holds the client connections and pushes. Designing the funnels, and sizing each to the volume it must handle, is the whole job. The streaming internals are taught in Kafka Streams, Stateful and Redis Cluster and Real-Time Fan-Out; this page owns the end-to-end pipeline.
Clarify requirements and pin the numbers
Functional requirements. The system ingests raw market-data quotes from exchanges and vendors for thousands of instruments. It computes derived analytics per instrument, including yield to maturity for bonds and intraday chart data for all instruments. It streams each client only the instruments they are currently viewing, in real time. It backfills the recent history on a client’s first subscription, so the chart and the current value appear immediately.
Non-functional requirements. Latency from quote to client must be low, sub-second for the live view, because the customer is watching. Throughput must absorb peak quote volume during volatile sessions. Scalability must reach millions of concurrent client connections. Correctness here is eventual consistency: the client sees a recent value, and the system tolerates brief gaps on reconnect by backfilling from the retained history.
Pin the client side. Assume one million concurrent WebSocket connections at peak. Each client views, on average, one or two instruments at a time, so the delivery tier is fanning perhaps two million instrument-subscriptions across the fleet, heavily concentrated on popular instruments. A hot stock may have hundreds of thousands of concurrent subscribers, which is the load that forces Sharded Pub/Sub rather than plain Pub/Sub.
WebSocket state cost, as in any million-connection system, is the hidden budget. Roughly a hundred kilobytes per connection means about a hundred gigabytes of memory fleet-wide just to hold the sockets. This number decides the pod count and is the reason the single-connection-per-instance rule is non-negotiable: the Redis subscription count must track the pod count, not the client count.
High-level architecture
The pipeline has four stages, each a funnel, and the boundaries are where the volume drops and the semantics change.
The ingest stage receives raw quotes from external feeds, normalizes them into a common envelope keyed by instrument id, and publishes to a Kafka raw-quotes topic partitioned by instrument. Normalization is its own concern because exchanges disagree on formats, timestamps, and symbol identifiers, and the documented candidate failure of assuming all exchanges provide consistent data formats lives here. The ingest stage is the only place that touches vendor quirks.
Normalization depth: the vendor adapter layer
Each exchange or vendor feed arrives in its own format. Some provide FIX protocol messages; some provide JSON over WebSocket; some provide CSV over SFTP. The vendor adapter layer translates each into the internal quote envelope, a Protobuf message with fields for the instrument identifier, the price, the exchange timestamp, a source sequence number, and a quality flag. The adapter is a separate deployment unit per vendor, so a format change at one vendor does not affect the others.
The adapter’s hardest job is timestamp normalization. Exchange timestamps are in different time zones, have different precision (some to the microsecond, some to the second), and some are the exchange’s system timestamp while others are the trade timestamp. The adapter converts every timestamp to UTC with millisecond precision and records the original timestamp in a metadata field so the downstream can audit. A timestamp that cannot be parsed is set to the adapter’s receive time and flagged, so the compute tier can decide whether to use it or reject it.
The adapter also handles instrument identifier mapping. Each vendor uses its own identifier scheme: some use ISIN, some use Bloomberg tickers, some use exchange-specific codes. The adapter maps the vendor identifier to the canonical ISIN using a reference-data lookup. The mapping is cached locally with a TTL and is refreshed on cache miss from the reference-data service. A mapping failure, where the vendor identifier is unknown, is logged and the quote is dropped because an unmapped instrument cannot be streamed. The drop rate is monitored as a signal that the reference-data mapping is stale or the vendor has introduced a new instrument.
The normalized quote envelope carries a source sequence number that is monotonic within the vendor’s stream. The compute tier uses this sequence to detect gaps and out-of-order arrivals. A gap in the sequence from a vendor triggers an alert because it may indicate a dropped quote that the vendor has not resent. Out-of-order quotes within a small window, a few hundred milliseconds, are reordered by the compute tier’s buffer; quotes outside the window are processed in arrival order and the sequence gap is logged.
The compute stage is a Kafka Streams topology that consumes the raw-quotes topic, computes per-instrument analytics, and writes results to two sinks: a Redis cluster for live streaming, and a Kafka computed-analytics topic for historical chart persistence. YTM is the published example; the same topology computes intraday chart points, moving averages, and any other derived view. Stateful aggregation in Kafka Streams holds the per-instrument state needed for the computation in a local RocksDB store, so the compute is local and fast.
The fan-out stage is the Redis cluster with Sharded Pub/Sub. Each computed analytic for an instrument publishes to a Redis channel keyed by instrument and analytic. Sharded Pub/Sub routes the channel to a shard, so the publish load spreads across the cluster rather than replicating to every node, which is what plain Pub/Sub would do. The fan-out tier is where the hot-instrument concentration is absorbed.
The delivery stage is the Ktor WebSocket fleet on Kubernetes. Each pod holds thousands of client WebSocket connections and maintains one Redis connection that subscribes to the channels its clients care about. Inside the pod, a Kotlin SharedFlow per channel fans each update to every connected client watching that instrument. On first client interest in an instrument, the pod subscribes to the channel; on last client departure, it unsubscribes, keeping the system lightweight.
The four funnels are summarised in the end-to-end architecture diagram.
The architecture’s elegance is that each stage’s cost is bounded by a different quantity. Ingest cost is bounded by quote volume. Compute cost is bounded by instrument count, not client count. Fan-out cost is bounded by the pod count times the channel count. Delivery cost is bounded by the client count. Because each stage scales independently, the system scales to millions of clients without any stage seeing million-scale load it cannot shed.
Deep dive: the Kafka Streams compute tier
The compute tier is where raw quotes become the analytic the client wants, and Kafka Streams is the right tool because the computation is stateful and per-instrument. YTM, for example, depends on the bond’s coupon, maturity, and current price; the price updates continually, but the coupon and maturity are static reference data. The topology joins the raw quote stream against the reference data, applies the YTM formula, and emits a new YTM value whenever it changes meaningfully.
State lives in a local RocksDB store keyed by instrument id, so the computation reads and writes locally without a network round trip. This is the central advantage of Kafka Streams for this tier: the state is co-located with the processor, which makes the per-instrument computation fast and the scaling clean. Adding partitions adds parallelism, because Kafka Streams assigns partitions to tasks and each task owns its slice of the state store. The deeper mechanics of stateful stream processing are taught in Kafka Streams, Stateful.
The double-sink pattern is important and is straight from the published architecture. Each computed analytic is written both to Redis, for the live stream, and to a Kafka topic, for the historical chart. The Redis sink serves the client watching now; the Kafka sink persists the history so a client arriving later can backfill the chart. Splitting the sinks lets each serve its purpose without compromise: Redis is optimized for low-latency fan-out, Kafka for durable retention.
Downsampling is a compute-tier responsibility, not a delivery-tier one. A chart does not need every quote; it needs a point per minute, or per second during volatility. The topology aggregates raw quotes into chart buckets and emits only the bucket summary, which is far less volume than the raw stream. Doing this in the compute tier means the fan-out and delivery tiers never see the raw volume, which is what makes the downstream stages affordable.
The chart history persistence and backfill protocol
The computed-analytics Kafka topic retains every chart point the topology produces. The retention is set to a value that covers the longest expected backfill window, typically several days. When a client subscribes to an instrument after having been disconnected for an hour, the delivery tier reads the chart points from the Kafka topic for the gap period and sends them to the client as a historical burst, followed by the live stream from that point forward.
The backfill protocol has three steps that the Ktor pod executes. First, on the client’s WebSocket subscription, the pod queries the chart history from the Kafka topic for the instrument between the client’s last known timestamp and now. The query is a Kafka consumer that reads the computed-analytics topic from a specific offset. Second, the pod sends the historical points to the client in chronological order. Third, the pod subscribes to the Redis channel for the instrument and streams subsequent updates.
The backfill latency matters because the client sees a blank chart until it completes. If the history is large, hours of chart points can be megabytes of data, which takes seconds to transfer over a mobile connection. The mitigation is to downsample the history at read time: send one point per minute for the oldest portion and one point per second for the most recent minute. The client renders the downsampled history immediately and then receives the live stream at full resolution. The downsampling is a policy decision that trades historical fidelity for perceived latency, and the policy parameters are exposed per instrument and per client connection speed.
### State store sizing and RocksDB tuning
The Kafka Streams state store in the compute tier is a local RocksDB instance per task. The state holds the per-instrument reference data, the current price, the chart bucket accumulators, and any intermediate values needed for the analytic computation. For thousands of instruments, the state store size is modest, on the order of hundreds of megabytes, and the write rate is the quote rate, which is the ingest rate.
RocksDB tuning for this workload follows a known configuration. The block cache is sized to hold the working set of hot instruments, typically a few hundred megabytes. The write buffer is sized to absorb burst writes during volatile sessions without flushing to SST files too frequently. The bloom filter is configured for point lookups because the state store is keyed by instrument id and accessed by key. The compaction style is level compaction, which keeps the write amplification low for the steady write rate.
The state store’s changelog topic is the durability mechanism. Every state change is written to the changelog, and on recovery, the state is rebuilt by replaying the changelog. The changelog topic must have sufficient retention to cover the longest expected recovery window, which is the time to replay the full changelog from the beginning. For a state store that has been running for days, the changelog can be gigabytes, and replay takes minutes. The mitigation is to take periodic snapshots of the state store and to compact the changelog topic so that only the latest value per key is retained after the snapshot.
A senior responsible for the compute tier monitors three RocksDB metrics above all others: the write stall count, which indicates the write buffer is filling faster than it can flush; the compaction pending bytes, which indicates backlog in the compaction pipeline; and the block cache miss rate, which indicates the working set is larger than the cache. A rising write stall count is the most urgent signal because it means the compute tier is falling behind the quote rate and will start dropping quotes.
The cold chart store for deep history
The Kafka topic retains only a few days of chart data. For the customer who opens a chart after a month away, the system needs a cold store that holds the complete history for every instrument. The cold store is a time-series database such as S3 with Parquet files, partitioned by instrument and date. When a client requests history beyond the Kafka topic’s retention, the delivery tier queries the cold store, merges the result with the Kafka topic’s recent data, and returns the merged range.
The cold store query is outside the forty-millisecond budget because it can take hundreds of milliseconds to read from S3 and parse the Parquet file. The response is asynchronous: the pod sends the live value and the most recent history from Kafka immediately, and the deep history arrives as a separate message when the cold store query completes. The client renders the immediate data first and fills in the deep history when it arrives, which gives the user a visible chart within seconds and a complete chart within a few seconds more.
The cold store is populated by a separate Kafka consumer that reads the computed-analytics topic and writes every chart point to S3, partitioned by date. The consumer batch-writes Parquet files every few minutes, trading freshness for write efficiency. The batch size is tuned so that a single consumer can keep up with the chart-point rate for all instruments. For a broker with thousands of instruments and one chart point per minute per instrument, the write rate is tens of chart points per second, which is easily handled by a single consumer writing batched Parquet files.
A common mistake is to compute the analytic inside the WebSocket pod, per client. That multiplies the computation by the client count watching the instrument and destroys the funnel property. Compute once per instrument in Kafka Streams, publish the result, and let the delivery tier only fan out the finished value.
Deep dive: Redis Sharded Pub/Sub and the WebSocket fan-out
The fan-out tier is the part that makes million-client delivery affordable, and its design rests on three rules, each load-bearing.
The first rule is Sharded Pub/Sub, not plain Pub/Sub. Plain Redis Pub/Sub broadcasts every published message to every node in the cluster, which means adding nodes to handle more clients increases the per-node message load rather than spreading it. At million-client scale with hot instruments, plain Pub/Sub saturates every node. Sharded Pub/Sub routes each channel to a specific shard based on a hash of the channel key, so the publish load for different instruments spreads across the cluster. This is precisely why Trade Republic chose Sharded Pub/Sub for the YTM pipeline, and the same reasoning applies to every market-data channel.
The second rule is one Redis connection per pod, not per client. Each Ktor pod subscribes, through a single Redis connection, to the channels its clients care about. When a message lands on a channel, the pod receives it once and fans it to every locally-connected client watching that instrument. This caps the Redis subscription count at the pod count times the channel count, never the client count. With a thousand pods and a few thousand active channels, that is millions of subscriptions manageable; with one subscription per client, it would be millions of connections and would melt Redis.
The third rule is the SharedFlow fan-out inside the pod. Kotlin SharedFlow is a hot flow with multiple collectors: one logical stream per channel, and every connected client watching that instrument collects from the same flow. The flow is created when the first client subscribes to the instrument and torn down when the last client leaves. This is the on-demand subscription pattern from the published architecture, and it keeps the pod lightweight: an instrument no one is watching costs nothing.
The three rules together make the fan-out tier elastic. Adding pods scales the delivery capacity without increasing Redis subscription count per pod, and adding instruments only creates new channels when clients are actually watching. The Redis connection per pod and the SharedFlow per channel together keep the per-instance cost proportional to the number of active instruments on that pod, never the global count.
The WebSocket connection lifecycle and reconnection protocol
A WebSocket connection between the mobile client and the Ktor pod goes through several states. The client initiates the connection with an authentication handshake that sends a JWT in the query string or the first message. The pod validates the JWT, associates the connection with the user, and returns a confirmation. The client then sends a SUBSCRIBE message with the list of instrument ids it wants to watch. The pod creates the Redis subscriptions on demand and starts streaming.
When the connection drops, because the mobile network is lossy, the client reconnects with a RECONNECT message that carries the last sequence number it received. The pod looks up the instrument subscriptions from a server-side session store keyed by the user id and recreates the subscriptions on the new connection. The session store is in Redis, so a reconnect to a different pod is transparent: the new pod reads the user’s subscriptions from the session store and recreates the SharedFlow and Redis subscriptions. The session store has a TTL, so a user who does not reconnect for an hour loses their subscription state and starts fresh.
The reconnection protocol must handle the race where the client sends a new SUBSCRIBE for an instrument while the server is still processing the previous RECONNECT for the same instrument. The server deduplicates subscriptions per instrument: if a subscription already exists for the user on this pod, a duplicate SUBSCRIBE is a no-op. The dedup is a simple set of active instrument ids per user in the pod’s memory, and it prevents duplicate Redis subscriptions and duplicate SharedFlow collectors.
Pod scaling and connection draining
The Ktor WebSocket fleet scales with the connection count. Each pod has a maximum connection limit, typically a few thousand, beyond which it refuses new connections and the load balancer routes new clients to other pods. The limit is set so that a pod has enough CPU and memory headroom to handle the fan-out work, the SharedFlow buffers, and the Redis subscription, without swapping.
When a pod is scaled down or redeployed, it must drain its existing connections before exiting. The pod enters a draining state where it refuses new connections but keeps existing ones open. It waits for the in-flight SharedFlow buffers to flush, which is typically sub-second, and then sends a DRAINING message to each connected client indicating that the connection will close. The client reconnects to another pod. The draining timeout is a few seconds: after the timeout, the pod forcefully closes all connections. The client reconnection is transparent because the reconnection protocol with the session store ensures the new pod knows the user’s subscriptions.
On-demand subscription is the detail that makes the system lightweight at the long tail. Most instruments have few or zero live viewers at any moment. If every pod subscribed to every channel eagerly, the Redis subscription count would be the pod count times the full instrument count, which is wasteful. Instead, a pod subscribes to a channel only when its first local client opens that instrument’s detail page, and unsubscribes when its last local client leaves. The net effect is that Redis holds subscriptions only for instruments with live interest, which is a small fraction of the catalog.
Data model
The data model is minimal because the pipeline is a stream, not a store. The key entities are the quote envelope, the computed analytic, and the chart bucket, and each flows through the pipeline as a Kafka message or a Redis publication.
The computed analytic’s key is the instrument id plus the analytic kind, which is also the Redis channel key. This alignment is deliberate: the same key identifies the data in Kafka, in Redis, and in the client’s subscription, so routing is uniform across the pipeline.
Tradeoffs and alternatives
A stateless consumer instead of Kafka Streams is the common alternative. A stateless consumer reads raw quotes and computes the analytic on each message without retaining state. This works only for stateless analytics, like a price percentage change, and fails for stateful ones like YTM, which depends on static reference data, or a chart bucket, which accumulates over a window. Kafka Streams with a local state store is the production choice because most useful market-data analytics are stateful, and the local store makes the computation fast and the scaling clean. The cost is operational complexity in managing the state store and rebalancing on partition changes.
Server-sent events instead of WebSockets is an alternative for the delivery tier. SSE is simpler, being unidirectional over HTTP, and works through more proxies. WebSockets are bidirectional, which the system needs because the client sends subscription and unsubscription commands. For a product where the client dynamically selects instruments, WebSockets are the right choice. SSE would force a new connection per instrument or a workaround, which is worse. See Redis Cluster and Real-Time Fan-Out for the WebSocket pattern.
Computing per-client in the delivery tier is the trap to avoid. It seems simpler to push raw quotes to the pod and compute there, but it multiplies the computation by the number of clients watching the instrument and destroys the funnel property that makes the system scale. Compute once per instrument in the streams tier; deliver the finished value.
Plain Redis Pub/Sub instead of Sharded Pub/Sub is simpler operationally but does not scale. The trade is operational maturity for capacity, and at million-client scale the capacity wins decisively. A pod that fans to thousands of clients receiving every market instrument via plain Pub/Sub would be saturated by instruments none of its clients watch.
A poll-based client instead of a push-based WebSocket is simpler and works through corporate proxies, but it imposes a latency floor equal to the poll interval and a load floor equal to the poll rate times the client count. For a real-time product, push is the product, with polling as a degraded fallback.
A single Redis cluster for all market-data channels versus a dedicated Redis cluster per market segment is an architecture question the Staff engineer may probe. A single cluster is easier to operate but mixes the load profiles of hot equity channels, cold bond channels, and high-throughput index channels. A hot equity channel that saturates its shard affects the latency of cold bond channels on the same shard. Dedicated clusters isolate the load profiles but increase operational surface and the total number of Redis connections the fleet must hold. The production answer depends on the scale: at Trade Republic’s volume, a single Redis cluster sized for the combined load is sufficient, with the hot-channel per-shard mitigation described above reserved for the few instruments that genuinely need it. A senior states the tradeoff and the scale at which they would move to dedicated clusters.
A Kafka Streams topology with a single sink versus the double-sink pattern is another tradeoff. A single sink to Redis leaves no durable history and makes backfill impossible. A single sink to Kafka makes live streaming through Redis impossible because Kafka consumers poll rather than subscribe. The double sink, writing to both Redis and Kafka, is the logical choice because the two outputs serve different purposes: Redis for push and Kafka for durable retention. The cost is that the topology writes two messages per computed analytic, which doubles the write throughput at the compute tier. For a system where the compute output is a fraction of the raw quote volume, the doubling is acceptable.
Ktor versus a dedicated WebSocket proxy such as Envoy’s WebSocket support is an infrastructure choice. Ktor provides fine-grained control over the WebSocket lifecycle: per-client backpressure, precise subscription management, and direct access to the Kotlin SharedFlow. A WebSocket proxy offloads connection management but cannot implement the on-demand subscription logic and the SharedFlow fan-out. At Trade Republic’s scale, the Ktor implementation is the correct choice because the subscription logic is integral to the delivery path. A proxy would add a hop without reducing complexity, because the proxy would still need to route to a service that manages the subscriptions.
Failure modes and what breaks at scale
A Redis cluster outage takes down the live stream. The graceful degradation is to fall back to a short-interval poll against a cached computed value, accepting higher latency, while the chart history in Kafka is unaffected and the compute tier keeps running. Because the compute tier writes to Kafka independently of Redis, no data is lost during a Redis outage; only the live delivery is degraded. This isolation between tiers is what makes the pipeline resilient.
A Kafka Streams rebalance, when a compute task moves to a new instance, briefly pauses the computation for the affected partitions. The local state store is rebuilt from the changelog topic, which takes seconds to minutes depending on the state size. During the pause, the Redis value for those instruments goes stale, but the chart history continues from the last computed value. The mitigation is to size the state stores so rebalance recovery is fast, and to monitor the rebalance duration. This is the operational cost of stateful stream processing.
A hot instrument concentrated on one Redis shard is the scaling limit of the fan-out tier. If one instrument has hundreds of thousands of subscribers and its channel hashes to one shard, that shard sees disproportionate publish and subscription load. The mitigation is shard-key design: for genuinely hot instruments, shard by subscriber-group within the channel, so the publish fans to multiple shards and each shard serves a subset of subscribers. This adds complexity and is reserved for the few instruments that need it.
Out-of-order or late quotes are a correctness concern at the ingest tier. Exchanges deliver quotes with jittery timestamps, and a late quote can cause the compute tier to emit a stale analytic on top of a newer one. The defense is to key by instrument and process in source-sequence order within a partition, and to use a grace period in windowed aggregations so late arrivals are handled rather than dropped. The documented candidate failure of assuming all exchanges provide consistent data formats and ordering lives here, and the normalizer plus sequence-aware processing is the answer.
A client reconnect after a drop must backfill the gap. On reconnect, the client sends the timestamp or sequence of the last value it rendered, and the pod queries the chart history from the Kafka topic or a downstream store to catch up, then resumes the live stream. Without backfill, a brief network blip looks like a gap in the chart, which erodes trust. The double-sink pattern, computed analytics to both Redis and Kafka, exists precisely to make this backfill possible.
A stale state store in Kafka Streams, when a RocksDB corruption occurs on the compute pod, can produce incorrect analytics for the affected instruments. RocksDB corruption happens rarely but is catastrophic when it does: the compute pod restarts, the state store is rebuilt from the changelog topic, and during the rebuild, the computed analytics for the affected partitions are not emitted. The mitigation is to monitor RocksDB health metrics and to have a standby compute pod that can take over with a warm state store. The standby pod consumes the same changelog topic and maintains a replica of the state store, so when the primary fails, the standby can start emitting within seconds rather than waiting for a full rebuild.
An out-of-memory condition on the Ktor pod, when the number of connected clients exceeds the pod’s capacity, can cause the pod to crash. The mitigation is strict per-pod connection limits enforced at the load balancer, memory-based autoscaling that adds pods when per-pod memory crosses a threshold, and a connection broker that monitors each pod’s connection count and routes new connections to the least-loaded pod. The connection broker is a separate service that the load balancer consults before routing a new WebSocket upgrade request, and it tracks the live connection count per pod in a shared Redis store.
Vendor feed failure and the fallback pricing strategy
When a primary vendor feed goes down, the system must fall back to a secondary vendor without a visible gap in the product. The ingest tier subscribes to two or more vendors for the same instrument and selects the primary feed as the authoritative source. When the primary feed stops producing quotes, the ingest tier switches to the secondary feed and publishes quotes from it with a quality flag indicating the fallback.
The switch is not instantaneous: the ingest tier detects the primary feed’s silence after a configurable timeout, typically a few seconds, and then starts publishing from the secondary feed. During the timeout, the compute tier continues to emit the last known value from the state store, so the client sees a stale but unchanging price. When the fallback kicks in, the price updates resume with a possible minor discontinuity if the two vendors have slightly different prices.
The fallback strategy requires that the secondary vendor’s quotes are normalized into the same envelope as the primary, which means the adapter layer for the secondary vendor must be maintained at the same quality as the primary. A senior does not treat the secondary vendor as a lesser system; it must be production-ready and monitored with the same rigor, because when it becomes the primary during an outage, any defect becomes immediately visible to all customers.
A vendor feed that returns to service after an outage may send a burst of stale quotes that span the outage period. The compute tier must ignore these stale quotes if they are older than a grace period, because applying them would cause the displayed price to jump backward in time. The grace period is set to the maximum expected processing delay plus a small buffer, and quotes outside the grace period are dropped. The dropped-quote rate is monitored as a signal that the vendor’s replay behavior is changing.
Deployment safety for the streaming pipeline
The streaming pipeline is a continuous system; stopping it means the client sees stale data. Deployments must be rolling and zero-downtime across all four tiers.
The ingest tier is stateless and deploys through a standard Kubernetes rolling update: new pods start, old pods drain, and no quotes are lost because the vendor connections reconnect to the new pods. The vendor adapter’s reconnection logic is idempotent: it starts from the last received sequence number and requests the vendor to replay from that point if the vendor supports it. Venn diagrams that do not support replay accept a small gap equal to the deployment window.
The compute tier, being stateful, must deploy through a rolling restart with cooperative rebalancing. Each compute task is migrated to the new version as the old pod is drained, and during the migration, the state store is transferred or rebuilt from the changelog. The deployment is orchestrated so that only a fraction of the tasks are migrated at a time, and the system sustains the quote rate with the reduced compute capacity. The deployment completes when every task has migrated to the new version.
The Redis cluster deploys through a rolling restart of its nodes, with each node failing over to its replica before restarting. The failover is transparent to the WebSocket pods: the pod’s Redis connection disconnects and reconnects to the replica, and the SharedFlow continues after a brief pause. The subscription state on the node is lost, so the pod must resubscribe to all channels after the reconnect. The resubscription is triggered by the Redis reconnect callback in the pod and happens within milliseconds.
The WebSocket fleet deploys through a rolling update with connection draining as described above. The key metric during a deployment is the WebSocket connection churn rate: a sudden spike in new connections signals that the draining is not working and clients are being disconnected prematurely. The deployment is paused if the churn rate exceeds a threshold, and the draining configuration is adjusted.
Data quality monitoring in the streaming pipeline
The streaming pipeline’s value depends on the quality of the quotes it ingests and the analytics it computes. A corrupt quote that passes through normalization and reaches the compute tier produces a bad analytic that the client sees. Data quality monitoring is the safety net.
The ingest tier validates each quote against a set of quality rules before publishing to Kafka. The rules include price reasonableness, the price is within a configured percentage of the previous price for the same instrument; timestamp freshness, the exchange timestamp is within a configured window of the ingest time; and source sequence monotonicity, the sequence number is within a small gap of the last sequence from the same source. A quote that fails any rule is rejected and logged, and the rejection rate per source is monitored. A sudden spike in rejections from one source signals a format change or a data quality issue at the source.
The compute tier also validates the analytics it produces. A computed YTM that is outside a reasonable range, for example, negative for a standard bond or above fifty percent, is flagged and not published to Redis or Kafka. The flagging is a circuit breaker: if the compute tier detects an anomalous analytic that is within the configured bounds but outside a dynamic range computed from the recent history, it logs the anomaly and continues with the previous value until the next quote produces a valid analytic. This prevents a single bad quote from producing a visible price spike on the client.
A senior instruments the data quality monitoring with a dashboard that shows per-source rejection rates, per-instrument anomaly rates, and the end-to-end health of the pipeline. The dashboard is the first thing checked during a customer-reported price inaccuracy, because it answers the question “did the pipeline ingest and compute the data correctly” before the investigation moves to the delivery tier.
Operational concerns
Observability is per-stage. At ingest, track the quote rate per vendor and the lag between exchange time and ingest time. At compute, track the streams task lag and the rebalance duration. At fan-out, track the Redis publish rate per shard and the shard hotness, the ratio of the hottest shard’s load to the average. At delivery, track the WebSocket connection count, the per-pod subscription count, and the end-to-end latency from quote to client.
Backpressure handling is tier-specific. The ingest tier must not let a slow vendor block a fast one, so each vendor has its own queue. The compute tier relies on Kafka consumer lag as the backpressure signal; if lag grows, add partitions or compute capacity. The delivery tier’s backpressure is the client: a slow client cannot block the SharedFlow, so slow clients are dropped or sampled rather than allowed to accumulate updates, because a stale update is worthless once a newer one exists.
Capacity planning keys off three numbers: the quote rate, which sizes ingest and compute; the active instrument count, which sizes the Redis channels; and the concurrent client count, which sizes the WebSocket fleet. The three scale independently, which is the pipeline’s core advantage. A common error is to size the whole system off the client count and then be surprised when a quote burst saturates the compute tier.
Cost management at million-client scale is mostly the WebSocket fleet. Holding a million connections is expensive in memory and in TLS handshake throughput at connect time. Connection draining and graceful reconnect during deploys, plus TLS session resumption, are the operational levers that keep the fleet affordable. The Redis cluster is comparatively cheap because the subscription count tracks the pod count, not the client count.
The TLS cost at connect time is a significant factor in fleet sizing. A new TLS handshake for each WebSocket upgrade uses asymmetric cryptography that is CPU-intensive. At a million clients reconnecting simultaneously after a deployment, the handshake burst can saturate the pod’s CPU and delay the WebSocket upgrade. The mitigation is TLS session resumption using session tickets: the server issues a session ticket to the client during the initial handshake, and the client presents the ticket on reconnect, which allows the server to resume the session with a symmetric-key exchange instead of a full handshake. Session resumption reduces the CPU cost of a reconnect by an order of magnitude.
The Redis cluster cost is driven by the shard count and the memory per shard. Each shard holds a fraction of the channels and a fraction of the subscription state. The memory per shard is proportional to the number of channels it owns, each channel’s subscription count, and the message buffer. For a system with tens of thousands of channels and millions of subscriptions, the memory per shard is gigabytes, and the cluster is sized for the combined subscription state plus headroom. The cluster is scaled by adding shards, which spreads the subscription state and the publish load.
Per-instrument cost attribution is an operational concern when the streaming pipeline serves multiple products. Equities, bonds, ETFs, and indices each have different quote rates and client subscription counts. A single cost pool obscures which product segment drives the infrastructure bill. The compute tier’s metrics per instrument, the quote rate, the state store size, and the Redis publish rate, are tagged with the instrument’s product segment, so the team can attribute cost per segment and optimize the most expensive ones. This level of cost observability is what lets a senior justify an infrastructure investment to the product team.
Defending the design to a Staff engineer
A Staff engineer will probe the funnel property, and you should state it plainly. The pipeline is a series of volume-reducing stages: raw quotes in, per-instrument analytics out, only-watched-instruments delivered. Each stage’s cost is bounded by a different quantity, so the system scales to millions of clients without any stage seeing unbounded load. Compute is bounded by instrument count; fan-out is bounded by pod count times channel count; delivery is bounded by client count.
The Redis scaling defense is the three rules: Sharded Pub/Sub so publish load spreads across shards, one connection per pod so subscriptions track pod count not client count, and SharedFlow fan-out inside the pod so one channel serves many clients. Together they make million-client fan-out affordable, and they are straight from the published Trade Republic architecture.
The state defense is Kafka Streams with local state stores. Stateful analytics like YTM and chart buckets need per-instrument state, and co-locating the state with the processor makes the computation fast and the scaling clean. Adding partitions adds parallelism; rebalancing rebuilds state from the changelog. The cost is operational, but the alternative, computing per client in the delivery tier, does not scale.
The pipeline is a series of funnels: ingest normalizes raw quotes to Kafka, Kafka Streams computes per-instrument analytics with local state, Redis Sharded Pub/Sub fans to interested pods with one connection each, and SharedFlow fans to clients inside the pod, so each stage scales on a different bounded quantity and the system reaches millions of clients without any stage melting.
Common mistakes candidates make
- Computing the analytic per-client in the delivery tier. This multiplies computation by the client count and destroys the funnel. Compute once per instrument in Kafka Streams.
- Using plain Redis Pub/Sub. It replicates every message to every node and does not scale past a few nodes. Sharded Pub/Sub is the production choice for million-client fan-out.
- One Redis connection per client. That is millions of connections and the scaling cliff. One connection per pod, fanning through SharedFlow, is correct.
- Eagerly subscribing to every channel on every pod. Most instruments have no live viewers. Use on-demand subscription: subscribe on first client interest, unsubscribe on last departure.
- Assuming consistent quote formats and ordering from all exchanges. The documented failure mode. Normalize at ingest and process in source-sequence order within a partition.
- Single-sinking to Redis only, with no durable history. A client reconnecting after a drop cannot backfill. Write computed analytics to both Redis for live and Kafka for history.
- Forgetting downsampling. Pushing every raw quote to the chart floods the client. Downsample to chart buckets in the compute tier.
- Ignoring hot-shard imbalance. A hot instrument can overload one Redis shard. Reserve subscriber-group sharding within a channel for the few instruments that need it.
- Not monitoring RocksDB write stalls. A rising write stall count means the compute tier cannot keep up with the quote rate and will start dropping quotes. Monitor RocksDB write stall, compaction pending bytes, and block cache miss rate.
- Deploying the compute tier without a standby or changelog-based recovery. A RocksDB corruption or pod failure requires rebuilding the state store from the changelog, which takes minutes. Have a standby pod with a warm state store or accept the recovery duration.
- Treating the secondary vendor feed as a lesser system. When the primary feed fails, the secondary becomes the live source. It must be production-ready with the same monitoring, normalization, and quality rules as the primary.
- Not handling stale quotes after a vendor feed recovers from an outage. The vendor may replay stale quotes from the outage period, causing the displayed price to jump backward. Ignore quotes older than a grace period configured to the maximum expected processing delay.
What the interviewer asks next
Expect to be pushed on cost. The question: a million concurrent WebSockets is expensive; how do you keep it affordable? The answer is the single-connection-per-instance rule capping Redis subscriptions at the pod count, TLS session resumption to cut handshake cost, connection draining for graceful deploys, and sizing the fleet off the connection count with headroom for bursts. The Redis cluster is comparatively cheap because the fan-out tier never sees the client count.
A second follow-up: what happens when Kafka Streams rebalances? The answer is that the affected partitions’ computation pauses while the state store rebuilds from the changelog, the Redis values for those instruments go briefly stale, and the chart history continues from the last computed value. The mitigation is sizing state stores for fast recovery and monitoring rebalance duration.
A third, deeper probe: a single instrument has half a million live subscribers and its channel overloads one Redis shard. What do you do? The answer is to shard within the channel by subscriber-group: the publisher fans the instrument’s updates to multiple channel keys, each hashing to a different shard, and each pod subscribes to one group. This spreads the hot instrument’s load across the cluster. It adds routing complexity and is reserved for the few instruments that need it, but it is the scaling lever when a single shard is the bottleneck.
Mastery Questions
Question
Why does the compute tier use Kafka Streams with a local state store rather than a stateless consumer?
Answer
Most useful market-data analytics are stateful: yield to maturity depends on static reference data, and chart buckets accumulate over a window. A stateless consumer cannot compute these. Kafka Streams co-locates a local RocksDB state store with the processor keyed by instrument, so the computation reads and writes locally without a network round trip, and scaling is clean because adding partitions adds parallelism with each task owning its state slice.
Sources & evidence5 claims · 2 cited
Design based on Trade Republic published streaming architecture (src_tr_ytm) and interview context (src_tr_interview); design reasoning beyond sources is internal-reasoning and carries no claim.
- Trade Republic's real-time market data streaming pipeline uses a Kafka Streams topology that consumes raw quotes, computes per-instrument analytics including yield to maturity, and sinks results to both a Redis cluster with Sharded Pub/Sub for live streaming and a Kafka topic for historical chart persistence.verified
- Redis Sharded Pub/Sub is used instead of plain Pub/Sub because plain Pub/Sub broadcasts every message to every Redis node and does not scale past a few nodes; Sharded Pub/Sub routes each channel to a specific shard so the publish load spreads across the cluster.verified
- A single Redis connection per Ktor service instance fans updates to all connected WebSocket clients regardless of client count; on the first WebSocket subscription for an instrument a Redis subscription is created and torn down when the last client leaves, keeping the system lightweight.verified
- The pipeline is a series of volume-reducing funnels: raw quotes in, per-instrument analytics out, only-watched-instruments delivered, so each stage's cost is bounded by a different quantity and scales independently.verified
- A common candidate mistake in market data streaming is computing the analytic per-client in the delivery tier, which multiplies computation by the client count and destroys the funnel property; the correct design computes once per instrument in Kafka Streams.verified
Cited sources
- Real-Time Bond Yield to Maturity: Streaming with Kafka and Redis · Trade Republic Engineering (Luis Jacintho)
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)