On this path · Data Infrastructure 4. Redis Cluster, Pub/Sub, and Real-Time Fan-Out
Redis Cluster, Pub/Sub, and Real-Time Fan-Out
Sharded Pub/Sub, single-connection multiplexing, and the SharedFlow fan-out pattern.
Learning outcomes
A customer opens the bond detail screen and watches the yield to maturity update as quotes arrive. Behind that single live number is a fan out problem with a nasty shape: thousands of instruments, each emitting updates, and thousands of customers, each watching a different subset, connected over WebSockets that can open and close at any moment. Serving that from the compute layer that produces the yields would couple the producers to every viewer and collapse under connection count. At Trade Republic the design that holds this together is a Redis Cluster with sharded Pub/Sub and a deliberately minimal connection model, and the engineering blog documents the reasoning in detail.
After studying this page, you can:
- Explain why a fan out problem at the WebSocket edge is fundamentally about multiplication of connections, and why the producer must not know about viewers.
- Describe Redis Cluster’s hash slot model, contrast it with consistent hashing, and explain why a publish or a script touches exactly one shard.
- Justify sharded Pub/Sub over regular Pub/Sub by the broadcast cost that destroys scalability in the non sharded model.
- Reason about the one Redis connection per service instance pattern, and why it scales independently of the WebSocket client count.
- Design a demand driven subscription lifecycle: subscribe on the first viewer for an ISIN, tear down when the last viewer leaves.
- Connect Kotlin SharedFlow and the hexagonal Port and Adapter architecture to the fan out, and describe the connect handshake of latest then stream.
- Defend the design under interview pressure and name the mistakes that expose a junior model.
Before we dive in
Suppose the Kafka Streams topology that computes yield to maturity wrote each result directly to every connected WebSocket. The first thing that would break is the producer: it would need to know which customers care about which ISINs, and the moment a customer opens or closes the app, the producer’s view of the world changes. The producer is now coupled to the viewer set, which moves constantly, and any spike in connections becomes a spike in producer work. The second thing that would break is scale: one producer instance cannot hold tens of thousands of open WebSockets, and spreading producers means deciding who owns which connection. The fundamental issue is that producing a value and fanning it to viewers are two different problems with two different scaling axes, and conflating them couples the fast axis, quote rate, to the slow but huge axis, connection count.
The role that separates these axes is a broker that is good at one thing: receiving a value for a channel and forwarding it to whoever is subscribed to that channel, without the sender knowing or caring who the subscribers are. That is what Pub/Sub is, and Redis is a fast, in memory implementation of it. The engineering blog’s yield pipeline sinks computed yields to Redis, and the WebSocket edge subscribes to Redis channels per ISIN. The producer writes once; the broker fans out; the edge holds the connections. Each layer scales on its own term.
But running Redis as a broker at scale hides a trap that the blog calls out explicitly. If you use Redis Cluster, and you use regular, non sharded Pub/Sub, every publish is broadcast to every node in the cluster, which destroys the scalability you adopted a cluster to get. The distinction between sharded and non sharded Pub/Sub, and the connection model around it, is the heart of this concept.
The fan out problem at the WebSocket edge
Let us be precise about the shape of the problem. The yield stream emits, for each ISIN, a sequence of yield updates. Each customer connected over a WebSocket is watching some set of ISINs, usually one or a handful. When a yield update for an ISIN arrives, every customer currently watching that ISIN must receive it, ideally within a few tens of milliseconds. The connection set is volatile: customers open the app, switch screens, close the app, all the time. The update rate is bursty: a quiet market emits little, an active market emits floods.
The design goal is that the cost of producing a yield update is independent of how many customers watch it. One write to the Redis channel for ISIN A delivers to ten customers or ten thousand customers with the same producer work. The multiplication happens inside the broker and the edge, on resources sized for connections, not on the compute that produces yields. This separation is the whole point of inserting Redis between the producer and the viewer.
Redis Cluster: hash slots, not consistent hashing
Redis Cluster shards data across multiple Redis nodes. The mechanism is hash slots, and it differs from the consistent hashing used by many distributed stores, so a senior engineer must keep them straight because they behave differently under resharding.
Redis Cluster defines 16384 hash slots. Every key maps to one slot by taking the CRC16 of the key and taking the remainder modulo the slot count. Each slot is owned by exactly one node in the cluster, so each key lives on exactly one node. To add or remove capacity, you move whole slots between nodes, and the slot ownership map is the cluster’s routing table, which every node knows so any node can redirect a client to the owner of a key.
Consistent hashing, by contrast, places both nodes and keys on a ring, and a key is owned by the next node clockwise. The practical difference surfaces when you resize. Consistent hashing moves only the keys between the new node’s position and its predecessor, a roughly proportional slice, and it does this without a fixed slot table. Redis Cluster moves whole slots, which is also bounded but explicit: you choose which slots to move, and the cluster tracks slot ownership centrally. The reason Redis uses a fixed slot count rather than a ring is operational: 16384 slots is small enough for every node to gossip the full map, which makes routing O(1) and rebalancing a matter of moving discrete slots rather than recomputing ring positions. The tradeoff is that the slot count is fixed at 16384, which bounds the cluster size in practice, but that bound is far above any realistic deployment.
A consequence a senior engineer must internalize is that a client can address any node and be redirected. When a client sends a command to a node that does not own the relevant slot, the node replies with a redirection, telling the client which node does own the slot. A smart client caches the slot map after the first redirection, so subsequent commands go straight to the owner. A naive client that does not cache pays a redirect on every command, which doubles latency. This is why production deployments use a cluster aware client library that maintains the slot map and refreshes it on cluster changes, rather than a plain connection that learns routing one redirect at a time.
Why sharded Pub/Sub, not regular Pub/Sub
This is the distinction the engineering blog emphasizes, and the one most candidates get wrong. Redis Pub/Sub has two modes in a cluster.
Regular, non sharded Pub/Sub broadcasts every published message to every node in the cluster. When a client publishes to a channel, the publishing node must forward the message to every other node, because a subscriber could be connected to any node. This means every publish costs a cluster wide broadcast, and every node receives every message regardless of whether it has a subscriber for that channel. As message rate grows, the broadcast traffic saturates the cluster’s inter node links and wastes work on nodes with no interested subscribers. The engineering blog is blunt: non sharded Pub/Sub hurts scalability because it broadcasts every message to every node.
Sharded Pub/Sub ties a channel to the same hash slot as its key. A channel named after an ISIN maps to the slot for that ISIN, owned by one node. Subscribers to that channel must connect to that specific node, and the publish goes only to that node, which forwards to its local subscribers. There is no cluster wide broadcast. The publish cost is proportional to the number of subscribers on that one node, not to the cluster size. This is the mode that scales, because each channel’s traffic stays on the shard that owns it.
The contrast is the whole argument for sharded mode: the left side pays a cost that grows with the cluster size on every publish, while the right side pays a cost that grows only with the subscribers on one shard. At Trade Republic’s message rate, the broadcast path is the one that breaks first, which is why the blog treats sharded Pub/Sub as non-negotiable.
A publish touches exactly one shard
Because a sharded channel maps to one slot and one node, a publish to it touches exactly that shard. The same is true for any operation that reads or writes a key: a GET, a SET, a Lua script, all go to the slot owner. This has a sharp consequence for multi key operations. A Lua script or a transaction that touches two keys in different slots will fail, because the cluster cannot route a single command to two nodes. To keep operations single shard, the keys involved must share a hash slot, which Redis enforces through hash tags: if a substring of the key is wrapped in braces, only that substring is hashed. Two keys like {isin:A}:latest and {isin:A}:meta share the slot because only isin:A is hashed.
The design lesson is to key your data and your channels so that everything for one entity lives on one shard. For the yield pipeline, the latest yield for an ISIN and the Pub/Sub channel for that ISIN should map to the same slot, which they do when the channel name derives from the ISIN. Then a publish and any associated state read or write touch exactly one node, with no broadcast and no cross shard coordination.
One Redis connection per service instance, many WebSocket clients
The connection model at the edge is the second half of the design, and it is where candidates often over engineer. The naive instinct is to open one Redis connection per WebSocket client, so each client has its own subscription. With thousands of clients that is thousands of connections per service instance, each a file descriptor and a buffer, and the resource cost scales linearly with the client count.
The engineering blog’s design does the opposite. Each service instance holds a single Redis connection that multiplexes subscriptions to all the WebSocket clients it serves, regardless of how many clients are connected. One connection, many logical subscriptions. The cost of the Redis connection is fixed per instance, decoupled from the client count, and the multiplication of viewers is handled in process, not across the network.
The reason this works is that a Redis subscription is a logical concept, not a physical connection. One connection can hold many subscribed channels, and messages arriving on that connection are dispatched to the right client set in process. The cost of adding a viewer is an in process operation, not a new network connection. This is the leverage that makes the design cheap to scale: doubling the customer count does not double the Redis connections, it only adds in process dispatch work that is already cheap.
Subscribe on first demand, tear down on last departure
Within that single connection, subscriptions are still managed per ISIN, and managed by demand. The engineering blog describes the lifecycle precisely: on the first WebSocket subscription for an ISIN, the service creates one Redis subscription to that ISIN’s channel. When the last client watching that ISIN leaves, the service tears the subscription down. In between, every new viewer for that ISIN is served by the existing subscription, with no new Redis work.
This keeps the system lightweight. If no one is watching ISIN A, there is no subscription for it and no Redis traffic. The moment the first viewer arrives, one subscription opens, and all subsequent viewers attach to it for free. When interest collapses to zero, the subscription closes and the work disappears. The subscription set tracks the actual demand, not the union of all instruments that might ever be watched, which is what keeps Redis load proportional to live interest rather than to the instrument universe.
SharedFlow: one ISIN, many viewers
The in process fan out from one Redis subscription to many WebSocket clients is implemented in Kotlin with SharedFlow, and the choice is worth understanding. A SharedFlow is a hot stream: it emits values to all current collectors, and collectors that join later do not replay past values, they only see future emissions. This matches the yield fan out exactly. One SharedFlow per ISIN collects from the Redis subscription for that ISIN, and every WebSocket client watching that ISIN collects from the same SharedFlow.
Two properties of SharedFlow are doing real work here. SharingStarted.WhileSubscribed connects the upstream subscription’s lifetime to the downstream collectors, so the Redis subscription opens when the first viewer arrives and closes when the last leaves, which is exactly the demand driven lifecycle. The replay of one retains the most recent yield so a newly connecting viewer gets the latest value immediately, before any new update arrives, which is the connect handshake of latest then stream expressed at the stream level.
A subtle but important distinction is between hot and cold flows, because it decides who pays for the upstream work. A cold flow, like the raw Redis subscription before shareIn, starts work afresh for each collector: if three viewers each collected it, three Redis subscriptions would open, defeating the whole design. shareIn converts it into a hot SharedFlow, which shares one upstream subscription among all collectors. The work of reading from Redis happens once, regardless of how many viewers collect, because the hot broadcast multiplexes the single upstream into many downstreams. This is the in process mirror of the Redis principle: one subscription feeds many viewers, and the cost is bounded by the channel, not by the subscriber count.
The fan out manager, which the core calls to obtain an IsinFanout for an ISIN, must also guard the lifecycle. When a viewer arrives for an ISIN with no existing fan out, the manager creates one, which opens the Redis subscription. When viewers arrive for an ISIN that already has a fan out, they simply collect the existing SharedFlow, paying nothing in Redis. When the last viewer leaves, hasObservers returns false, the manager cancels the fan out, and the Redis subscription closes. A bug here, where a fan out is retained after its last viewer leaves, leaks a Redis subscription and a SharedFlow per orphaned ISIN, which accumulates over days. The tear down on last departure is not an optimization; it is a leak prevention invariant.
The hexagonal port and adapter
The engineering blog notes that the service follows hexagonal architecture, the Port and Adapter pattern, and this is not incidental decoration. In hexagonal architecture the application core defines ports, which are interfaces describing what it needs to do, and adapters, which are the concrete implementations of those ports for a specific technology. The core never names Redis or Ktor or Kafka directly; it names abstractions like a market data port and a streaming port.
The value at the WebSocket edge is that the fan out logic is testable and replaceable. The core decides, given a new yield for an ISIN, to push it to the SharedFlow for that ISIN. The Redis adapter implements the actual subscription mechanics. The Ktor adapter implements the actual WebSocket transport. A test can substitute an in memory adapter for the Redis port and verify the fan out logic without a Redis instance, and a future migration to a different broker touches only the adapter, not the core. The decoupling is what lets the demand driven subscription lifecycle and the SharedFlow fan out be reasoned about independently of the transport.
The connect handshake: latest then stream
When a WebSocket connects for an ISIN, two things must happen, in order. First, the customer must see the current yield for that ISIN immediately, because a blank screen on connect is a poor experience. Second, the customer must then receive every subsequent update as it arrives. These have different data sources: the latest yield is a point lookup, and the stream is a subscription.
The latest value comes from a Redis GET on the key holding the current yield for that ISIN, written by the Kafka Streams sink. The stream comes from a Redis subscription to the channel for that ISIN. Sending the latest first and then subscribing avoids a race: if you subscribed first and then fetched the latest, an update could arrive between the two and be lost or duplicated. Fetching the latest, sending it, and then subscribing means the customer starts from a known value and every later update is delivered in order. The SharedFlow replay of one, described above, is the in process mechanism that makes this clean: the connecting collector immediately receives the most recent value held by the replay buffer, then continues with live emissions.
What breaks at scale and how to operate it
Every layer of this design has a failure mode the operator must understand.
The single Redis connection per instance is a bottleneck and a single point of failure for that instance. If the connection drops, every WebSocket client served by that instance loses its stream until the connection reconnects and the subscriptions are re established. The mitigation is to make reconnection and resubscription automatic, and to size instances so the loss of one sheds a bounded fraction of connections. A connection pool, rather than literally one connection, can raise the ceiling, but the principle stands: the connection count is fixed per instance, not per client.
Sharded Pub/Sub requires the channel to map to one slot, which means subscribers must connect to the node that owns the slot. A cluster resharding, when slots move between nodes for rebalancing, can briefly disrupt subscriptions on the moved slots. The operator must expect subscriptions to reconnect after a reshard, and the demand driven lifecycle means a disrupted subscription simply re opens when its viewers are still present.
Memory is the binding constraint. Redis is in memory, so the latest yields for the instrument universe and the in flight Pub/Sub messages live in RAM. A surge in watched ISINs or a spike in update rate grows memory use, and evictions or OOM are catastrophic. Monitoring memory, setting eviction policies deliberately, and sizing the cluster for peak interest, not average, are operational discipline. The latest per key is bounded by the instrument universe, which is large but finite; the Pub/Sub traffic is bursty and must be sized for the active market.
A word on eviction policy, because the wrong choice is silently destructive. Redis can evict keys by least recently used, by least frequently used, by time to live, or not at all. For the latest yield cache, the right choice is a policy that evicts cold ISINs, those no one is watching, when memory pressure rises, which is exactly what the demand driven lifecycle already encourages by tearing down subscriptions. The wrong choice is no eviction, which means a memory surge triggers OOM and the node crashes rather than shedding cold data. The discipline is to size the cluster so that the working set of hot ISINs fits comfortably, and to configure eviction so that the rare surge degrades gracefully by dropping the coldest keys, not by crashing the node and disconnecting every viewer on it.
The SharedFlow replay buffer is small but it holds a reference per ISIN with any viewer, and the subscription map holds an entry per watched ISIN. Leaks occur if a fan out for an ISIN is not torn down when its last viewer leaves, which is why the hasObservers check and the tear down on last departure exist. A bug that retains fan outs for departed viewers slowly leaks memory until the instance is restarted.
Defending the design in an interview
The interview questions probe whether you understand the separation of concerns, the hash slot model, and the connection economics.
Why insert Redis between the yield producer and the WebSocket? Because producing a value and fanning it to viewers are different problems on different scaling axes, and Redis is a fast broker that fans one write to many subscribers without the producer knowing the viewer set. Why sharded Pub/Sub and not regular? Because regular Pub/Sub broadcasts every publish to every cluster node, which saturates the cluster as message rate grows; sharded Pub/Sub routes the publish to the one node that owns the channel’s slot, so traffic stays local. Why one Redis connection per instance? Because a subscription is logical, not physical, so one connection multiplexes many channels and the connection cost is fixed per instance, decoupled from the client count. Why subscribe on first demand and tear down on last departure? Because it keeps Redis load proportional to live interest, not to the instrument universe. Why SharedFlow with replay one? Because it is a hot stream that fans one subscription to many collectors, and the replay delivers the latest value to a late joiner, which is the connect handshake. How are Redis hash slots different from consistent hashing? Slots are a fixed table of 16384 mapped by CRC16, moved in discrete units; consistent hashing is a ring where keys go to the next node clockwise. Both bound data movement on resize, but the slot table gives O(1) routing via a gossipped map.
Common mistakes candidates make
- Using regular Pub/Sub on a cluster. The broadcast to every node destroys the scalability the cluster was adopted for; candidates who do not know the difference pick the wrong mode.
- Opening one Redis connection per WebSocket client. This scales the connection cost with the client count, exactly the linear blowup the design avoids.
- Keeping a subscription alive with no viewers. Forgetting the tear down on last departure leaks Redis subscriptions and instance memory, growing load with dead interest.
- Conflating hash slots with consistent hashing. They differ in routing and resharding, and a candidate who treats them as the same misunderstands how the cluster routes.
- Publishing before fetching the latest on connect. Subscribing first and fetching second races updates, losing or duplicating the boundary value; the handshake is latest then stream.
- Forcing multi key operations across slots. A script or transaction touching keys on different shards fails, because the design must keep each entity’s data on one shard via hash tags.
- Coupling the producer to the viewer set. Letting the yield compute layer know about WebSockets couples the fast quote rate axis to the slow but huge connection axis, the exact conflation the broker exists to prevent.
- Treating Redis as durable. It is in memory; memory is the binding constraint, and OOM or eviction is catastrophic, so sizing for peak interest and monitoring memory is mandatory.
- Using a naive Redis client that does not cache the slot map. Every command to the wrong node pays a redirect, doubling latency, because the client relearns routing one miss at a time.
What an interviewer asks next
After the single connection answer, expect: if there is one Redis connection per instance, is that connection a single point of failure for the instance? The answer is yes, for that instance. If the connection drops, every WebSocket client served by that instance loses its stream until the connection reconnects and the subscriptions re establish. The mitigation is automatic reconnection, resubscription driven by the SharedFlow collectors that are still active, and sizing instances so one loss sheds a bounded fraction of connections rather than all of them. The demand driven lifecycle helps here: a re established connection only resubscribes to ISINs that still have viewers.
After the sharded Pub/Sub answer, expect: what happens to subscriptions during a cluster reshard, when a slot moves between nodes? The subscribers to that slot’s channels must reconnect to the new owner, because sharded Pub/Sub requires the subscriber on the owning node. During the brief window of the move, updates on the moved channel are missed. The design tolerates this because the demand driven lifecycle re opens the subscription on the new owner, and the connect handshake’s latest then stream refill means a reconnecting viewer immediately gets the current yield rather than a stale gap. A reshard is rare and bounded, so the occasional missed update is acceptable for a real time price stream.
After the hash slot answer, expect: how do you keep the latest yield and the Pub/Sub channel for one ISIN on the same shard? The answer is hash tags. The latest yield key and the channel name both wrap the ISIN in braces, so the CRC16 hashes only the ISIN, forcing both to the same slot and therefore the same node. Then a publish and a read of the latest value for one ISIN touch exactly one shard, with no broadcast and no cross shard coordination.
Mastery Questions
Question
Why does the yield pipeline insert Redis between the Kafka Streams producer and the WebSocket clients, rather than writing directly to connected clients?
Answer
Producing a value and fanning it to viewers are different problems on different scaling axes. If the producer wrote to clients directly, it would need to know the volatile viewer set, coupling the fast quote rate axis to the slow but huge connection count axis. Redis is a fast broker that fans one write to many subscribers without the producer knowing who they are. The producer writes once; the broker multiplies; the edge holds the connections. Each layer scales on its own term.
Sources & evidence6 claims · 1 cited
Six claims: five sourced from the Trade Republic yield-to-maturity post (Sharded Pub/Sub, single connection, demand-driven subscribe, SharedFlow, connect handshake), one stable-common-knowledge on Redis Cluster hash slots versus consistent hashing.
- Redis cluster with Sharded Pub/Sub is used because non-sharded Pub/Sub broadcasts every message to every node and hurts scalability.verified
- A single Redis connection per service instance fans updates to all connected WebSocket clients regardless of client count, optimizing resource use.verified
- On the first WebSocket subscription for an ISIN a Redis subscription is created; when the last client leaves it is torn down, keeping the system lightweight.verified
- Kotlin SharedFlow broadcasts one ISIN's updates to many WebSocket clients, and the service follows hexagonal architecture (Port and Adapter).verified
- Ktor WebSockets stream updates; on connect the latest YTM is sent immediately then subsequent updates stream.verified
- Redis Cluster maps a key to one of 16384 hash slots by CRC16 modulo the slot count; each slot is owned by one node, which is why a publish or script touches exactly one shard, unlike consistent hashing's ring model.stable common knowledge
Cited sources
- Real-Time Bond Yield to Maturity: Streaming with Kafka and Redis · Trade Republic Engineering (Luis Jacintho)