Kafka Streams for Stateful Transforms

Learning outcomes

A bond does not trade on price alone. A customer comparing two bonds wants to know the yield to maturity, the annualized return they would earn if they held the bond to its end date at the current price. That yield changes with every price quote, and quotes arrive continuously throughout the trading day. Computing it once and storing it is not enough, because the input moves and the output must move with it, in real time, for thousands of instruments, fanned out to thousands of customers. At Trade Republic this continuous transform is the job of a Kafka Streams topology, and the engineering blog documents the design in detail.

After studying this page, you can:

• Model a stream processing pipeline as a directed acyclic graph of processors, and explain why that shape fits a continuous transform.
• Distinguish a KStream, a stream of events, from a KTable, a stream reduced to current state, and choose between them per stage.
• Explain how a stateful transform is backed by an embedded state store whose durability comes from a Kafka changelog topic.
• Justify the dual sink, writing each computed yield to both Redis and a Kafka topic, by the downstream consumers each serves.
• Reason about offset commits, the commit interval, and graceful shutdown as the levers that trade latency against correctness.
• Defend in app stream processing over a micro batch alternative for a low latency, per key transform.
• Articulate the failure modes and the mistakes that expose a junior model under interview pressure.

Before we dive in

Imagine the simplest possible design for real time yield. An external service subscribes to a Kafka topic of raw bond quotes, computes the yield for each quote, and publishes the result. This works for a trickle of data. It collapses the moment the quote rate rises and the service cannot keep up, or the moment the service restarts and must decide where it left off, or the moment two instances both process the same instrument and disagree on the latest yield.

These are not three separate problems. They are three symptoms of one missing piece: state. To compute yield per instrument and to publish only the latest value, the processor must remember, per instrument, what it has already emitted. To resume after a crash without losing or duplicating work, that memory must be durable and tied to the input position. To scale horizontally without two instances fighting over the same instrument, that memory must be partitioned by the same key as the input. A stream processing framework’s reason to exist is to give you exactly this: state that is partitioned with the stream, durable through a changelog, and replayable from a known offset, all without you writing the machinery.

Kafka Streams is the library that runs this inside your application rather than as a separate cluster. It is not a standalone server you operate; it is a set of threads inside your service that consume Kafka topics, maintain local state stores, and produce Kafka topics, with the framework handling partition assignment, rebalancing, fault tolerance, and offset management. The engineering blog’s yield to maturity pipeline is built on it, and walking through that pipeline is walking through every concept that matters for stateful streaming.

From raw quote to yield: the problem that demands a topology

The input is a Kafka topic where each record is a raw quote for a bond, keyed by its ISIN. The output the business needs is, for each ISIN, the current yield to maturity computed from the latest quote, pushed to customers watching that bond and also appended to a historical series for charting. This is a per key transform: every quote for one ISIN produces a new yield for that ISIN, and the yields for different ISINs are independent.

A topology is the right shape because the work decomposes into a chain of independent steps, each with one job. Read the quote. Compute the yield from the quote’s price, coupon, and maturity. Drop quotes that yield no valid result. Fan the result to two sinks. Each step takes the previous step’s output as its input, and nothing loops back, which is exactly what a directed acyclic graph expresses. Modeling the pipeline this way makes each stage testable and replaceable, and lets the framework parallelize by partition across instances without you reasoning about concurrency.

The diagram is not decorative; it is a literal description of the processors the framework wires together. Source, mapValues, filter, and sink are the named processor types, and the edges are the subscriptions between them.

The topology as a directed acyclic graph

A Kafka Streams topology is a graph of processor nodes. A source node subscribes to one or more input topics. A stream processor node applies a transformation: it can read records from upstream, compute, and emit records downstream. A sink node writes records to an output topic. Edges are the flow of records from one node to the next. Because records only flow forward, the graph is acyclic, which is what makes the framework’s guarantees possible: it can reason about progress, commit offsets, and checkpoint state without detecting or breaking cycles.

The topology is built declaratively in code. You describe the graph using the StreamsBuilder DSL, and the framework compiles it into a set of tasks, one per partition of the input topic. Each task owns the processing for its partition and the state for the keys in that partition. When the service runs multiple instances, the framework assigns partitions across instances, so adding instances scales throughput up to the partition count, the same ceiling we saw in the Kafka concept. This is the deep consistency of the model: the partition is the unit of parallelism, of ordering, and of state ownership.

KStream and KTable: two views of a stream

A KStream is a stream of records, where each record is an independent event. A quote arriving is a KStream record. A KTable is a stream reduced to current state, where each record updates the value for its key, so the table holds only the latest value per key. A KTable is what you want when downstream logic needs the current state of an entity rather than the history of its changes.

The choice between them at each stage is a design decision driven by semantics. The raw quotes are a KStream, because each quote is a discrete event. The computed yields, before any reduction, are still a KStream, because each is a fresh transform of a quote. The latest yield per ISIN, the value customers watch, is conceptually a KTable, because only the current value matters. Whether you materialize it as a KTable depends on whether downstream needs the table abstraction or whether a simple sink to Redis, which itself holds latest per key, is enough. The yield pipeline sinks to Redis, so Redis plays the role of the materialized current state, and the Kafka topic of yields plays the role of the append only history.

Serdes and the cost of serialization

Every record that crosses a processor boundary, and every record written to or read from a state store, is serialized and deserialized by a serde, a serializer plus deserializer pair. The choice of serde is not free. A JSON serde is human readable but slow and large. An Avro serde is compact and schema aware but requires a schema registry. A Protobuf serde is compact and typed but requires generated classes.

The engineering blog’s outbox material records a deliberate choice: the payload is serialized on the application side before insertion, using Avro, and the schema is registered explicitly at deploy time, not at runtime. This fails fast on schema mismatch, which prevents poison pills at the database and capture layers, and it lets Debezium pass the already serialized bytes to Kafka without parsing. The same principle applies to the stream topology: choosing a compact, schema validated serde early, and registering schemas as part of the deploy, avoids a class of runtime failures that would otherwise surface as unprocessable records in the middle of a hot pipeline.

Stateful transforms and the state store

A stateless transform, like mapValues, depends only on the current record. A stateful transform depends on more than the current record: an aggregation, a join against a slowly changing dimension, a running count. Stateful transforms are where the framework earns its keep, because they require the processor to remember something across records, and that memory must survive crashes and respect partition ownership.

Kafka Streams backs every stateful transform with a state store, a local, embedded key value store that lives on the same machine as the task processing the partition. The store is partitioned with the stream: the task that owns partition three of the input topic owns the slice of state for the keys in partition three. Because partition assignment is exclusive within a consumer group, no two instances ever touch the same key’s state, which removes the coordination problem that a naive external cache would create.

Why an embedded store beats an external cache

The temptation, when a transform needs memory, is to reach for an external cache like Redis and read and write it from each record. That works and then degrades. Each record pays two network round trips, one read and one write, which adds latency that compounds across a long pipeline. The cache becomes a dependency that must itself be operated, monitored, and kept consistent with the stream. And because the cache is shared, two stream instances processing different partitions of the same key, after a rebalance, can race and corrupt the state.

The embedded state store removes all of this. State lives in a local RocksDB instance on the same disk as the task, so reads and writes are microseconds, not network calls. There is no shared external dependency for the in flight state. And because the store is partitioned with the stream, ownership is exclusive and there is no cross instance race. The cost is that each instance holds a slice of state, so when an instance fails its state must be rebuilt, which is where the changelog comes in.

The choice of RocksDB is itself a deliberate tradeoff. RocksDB is an embedded, log structured merge tree store that holds recent data in memory and older data on disk, spilling automatically as the store grows. For a state store that may exceed memory, this is the right shape: hot keys are served from the in memory memtable at memory speed, and colder keys are served from disk at the cost of a seek. A pure in memory store would cap the state at memory size and fail or evict when exceeded, which is unacceptable for a store that may grow with the instrument universe. By spilling to disk, RocksDB lets the state store scale beyond memory while keeping the common case fast, which is why the framework chose it as the default backing store.

Changelogs: the state store is just a Kafka topic

The durability of a state store comes from a Kafka changelog topic. Every write to the state store is also appended to a compacted changelog topic. If an instance fails, the framework reassigns its partitions to a surviving instance, which rebuilds the state store by replaying the changelog from the beginning. Because the changelog is partitioned by the same key as the input, the rebuild is local and parallel.

This is the elegant core of the model. The state store is not a separate system you operate; it is a local cache of a Kafka topic. The topic is the source of truth for the state, and compaction keeps it bounded to the latest value per key. Durability, ownership, and replay all flow from the same mechanism, the partitioned log, which is why the design is coherent where an external cache design would be a patchwork.

The yield topology, in Kotlin

The engineering blog describes the topology in Kotlin, running inside a service that also owns the Redis sink and the WebSocket fan out. The core is a StreamsBuilder that reads the quote topic, applies mapValues to compute the yield, filters out null results, and writes to both Redis and a yields topic.

Two details deserve attention. First, mapValues preserves the key, so no repartition occurs. Because the input is already keyed by ISIN, and the transform keeps that key, the records stay on the same partition through the whole chain. A repartition would write to an intermediate topic and re read, which adds latency and a second topic to operate; avoiding it is a real optimization the design earns by keying the input correctly from the start. Second, the Redis sink is attached by the caller, not hidden inside the builder, because the Redis write is a side effect that must be coordinated with the framework’s commit cycle, which is the subject of the next section.

Two sinks, two consumers

Each valid yield is written to two places, and the choice is deliberate, not redundant. The Kafka topic of yields is an append only history; a charting service reads it to reconstruct the yield series over time. Redis holds only the latest yield per ISIN, because the WebSocket fan out needs the current value the moment a customer connects, and a customer does not want the full history pushed to their screen. The same computed value serves two access patterns, history and latest, and routing them to two stores designed for each is cleaner than forcing one store to serve both.

Offsets, commits, and graceful shutdown

A stateful stream processor must manage two things atomically: the position in the input, the offset, and the side effects it has produced, the output records and the state store writes. If the processor crashes after writing output but before committing the offset, a restart reprocesses the input and duplicates the output. If it commits the offset before writing output, a crash loses output. Kafka Streams manages this by committing periodically: it flushes state store writes and output records to Kafka, then advances the committed offset, as one unit tied to the commit interval.

The commit interval is the lever that trades latency against overhead. A short commit interval commits more often, which lowers the bound on duplicates after a crash but adds more commit overhead. A long commit interval batches more work between commits, which is cheaper but means a crash reprocesses more. For a yield pipeline the default, around one hundred milliseconds, balances freshness of the output against commit cost, and the framework handles the flush and the offset advance together so the unit of work is consistent.

Graceful shutdown is not a nicety. When the service stops, in flight records must be drained, the state store flushed, and the offset committed, or the next start reprocesses work and duplicates output. The shutdown hook calls close with a timeout, giving the framework a bounded window to finish cleanly before the process is killed. Under a hard kill the framework relies on the commit interval and the changelog to recover, but a clean close is what makes deploys safe. The processing guarantee of exactly once, set here, makes the consume transform produce cycle atomic within Kafka, so even an unclean recovery does not duplicate output records.

Stateful in app processing versus micro batch

A common alternative to a continuous stream processor is a micro batch engine, which reads a window of records, processes them as a batch, and repeats on an interval. The question a senior engineer must answer is when each is the right choice, and the yield pipeline illustrates the answer.

Micro batch is the right tool when the transform is naturally analytic: an aggregation over a time window, a join across many sources, a query that benefits from staging data and running a computation over the whole stage. It adds latency equal to the batch interval, because results cannot appear until the batch closes. For a dashboard computing daily statistics, that latency is invisible.

The yield transform is the opposite case. Each quote must produce a yield within milliseconds, because a customer is watching the price move. There is no window to wait for and no aggregation across records; the transform is per key and immediate. A continuous, per record processor has the lowest possible latency because it emits as soon as a record arrives, with no batch boundary. Running it in app, as library threads, also removes the operational burden of a separate cluster to run and tune. The engineering blog’s choice of Kafka Streams is the choice of the tool whose model fits the workload: continuous, per key, low latency, with state managed by the log rather than re invented.

What breaks at scale: state, standbys, and the rebalance tax

A stateful stream processor’s failure modes all trace back to the state store and the rebalance, because those are the places where work in flight meets durable state.

The first is the state store rebuild, sometimes called warmup or restoration lag. When a task is assigned to an instance that does not yet hold its state, the instance must replay the changelog from its earliest retained offset to rebuild the store before it can process new records. For a small store this is fast, but for a large store it can take minutes, during which the partition is not processed and consumer lag grows. The mitigation is standby replicas, extra copies of the state store kept on other instances that are closer to current; on a rebalance the framework prefers an instance that already has a warm copy, shrinking the restore window. The tradeoff is memory: standbys multiply the state footprint.

The second is the rebalance tax on deploy. Every time you roll the service, instances leave and join, triggering rebalances that revoke and reassign partitions. During the revocation, in flight records are drained and state is checkpointed; during assignment, state is restored. A rolling deploy across many instances therefore causes a stutter, not a single pause, as each instance’s partitions migrate in turn. The discipline is to roll slowly, one instance at a time, so only a fraction of partitions migrate at once, and to keep the state store small enough that restore is quick.

The third is the dirty close. If an instance is killed without the shutdown hook, the last batch of processed records may not have had its offset committed, so the next owner reprocesses them from the last commit. With exactly once processing within Kafka, the output topic writes are deduplicated, but any external side effect, like the Redis sink, sees the duplicate and must be idempotent. This is why the Redis write in the yield pipeline keys its updates by ISIN, so a replayed yield overwrites rather than appends.

The fourth is the processing guarantee’s hidden cost. Exactly once processing, set via the transactional API, adds overhead to every commit, because the output writes and the offset commit must be enclosed in a transaction. For a hot quote stream the overhead is usually worth the correctness, but a senior engineer measures it and downgrades to at least once with idempotent consumers when the latency budget cannot absorb the transaction cost. The choice is never exactly once everywhere; it is exactly once where it matters and at least once where latency does.

Defending the design in an interview

The questions on stateful streaming probe whether you understand the topology as a graph, the state store as a projection of the log, and the commit cycle as the atomicity boundary.

Why a topology rather than a loop that reads and writes? Because the graph makes each stage independent, testable, and parallelizable by partition, and lets the framework manage offsets and state without you writing concurrency control. Why an embedded state store rather than Redis? Because local RocksDB reads are microseconds, there is no shared external dependency for in flight state, and ownership is exclusive per partition, removing cross instance races. How is the state store durable if it is local? Because every write is mirrored to a compacted changelog topic, and on failure a surviving task rebuilds the store by replaying the changelog. Why sink to both Redis and a Kafka topic? Because the topic is an append only history for charts and Redis is the latest per key for real time fan out, and the same computed value serves both. Why is graceful shutdown important? Because it drains in flight records, flushes the state store, and commits the offset, so a restart does not duplicate output or corrupt state.

Common mistakes candidates make

• Using an external cache for in flight state. Each record pays network round trips, the cache is a shared dependency prone to races, and durability is now your problem rather than the changelog’s.
• Forgetting the changelog. Assuming the local state store is durable on its own, when in fact durability comes entirely from the mirrored compacted changelog topic.
• Repartitioning unnecessarily. A mapValues that drops the key, or an early repartition, adds an intermediate topic and latency that correct keying would have avoided.
• Choosing the wrong processing model. Reaching for micro batch when the workload is per key and low latency, adding an invisible but real batch delay.
• Ignoring the commit interval. Setting it too high and accepting large duplicate windows after crashes, or too low and paying commit overhead on a hot stream.
• Skipping the shutdown hook. Hard kills that leave uncommitted work and force replays, because the framework never got to drain and commit.
• Overstating exactly once. Claiming the Redis sink participates in the Kafka transaction, when in fact the guarantee holds within Kafka and the Redis write is still at least once.
• Treating the topology as a black box. Not reasoning about per stage KStream versus KTable, which leads to holding history when current state was wanted, or vice versa.
• Sinking history into the state store. Letting the store grow unbounded by retaining per key history, when history belongs in an output topic and the store should hold only current state.
• Forgetting that standby replicas cost memory. Enabling standbys for faster restore without accounting for the multiplied state footprint across instances.

What an interviewer asks next

After the state store answer, expect: what happens to in flight records when a rebalance strips a partition from an instance? The instance must stop processing, flush any pending writes to its state store and output topics, and commit its offset before revoking, so the new owner resumes from a consistent point. The framework manages this, but the application must not hold work outside the framework’s view, for example a pending Redis write on a separate thread, because that work is invisible to the commit and can be lost or duplicated.

After the changelog answer, expect: how do you size the state store, and what happens if it grows beyond memory? The store is backed by RocksDB, which spills to disk, so a store larger than memory is supported but slower, because reads incur disk seeks. The discipline is to keep the store to current state only, sink history to a topic, and enable compaction on the changelog so it stays bounded to the latest per key. A store that holds history grows without limit and turns every rebalance into a long restore.

After the dual sink answer, expect: why not just write to Redis and let the chart service read Redis? Because Redis holds only the latest value per key, which is the wrong shape for a historical chart that needs the sequence of values over time. The chart service needs an append only log, which is exactly what a Kafka topic is. Forcing Redis to serve history would mean either polling it on an interval, which adds latency and load, or extending it with a list per key, which fights its key value model. Two stores for two access patterns is the cleaner design.

Mastery Questions