On this path · Data Infrastructure 2. Kafka as the System of Motion
Kafka as the System of Motion
Partitions, key ordering versus offset ordering, consumers, and exactly-once semantics at Trade Republic scale.
Learning outcomes
A brokerage has a source of truth for money in its database, and it has dozens of services that must react to every change in that truth: the notification service when an order fills, the analytics service when a price moves, the tax engine when a dividend lands. Moving those events reliably, in order, at the volume a brokerage generates, is the job of the streaming backbone. At Trade Republic that backbone is Kafka, and the engineering blog treats it not as a message queue but as the system of motion: the committed log that every downstream projection subscribes to.
After studying this page, you can:
- Explain why a partitioned append only log is the right spine for event motion, and why ordering is guaranteed within a partition but not across them.
- Predict how a message key routes to a partition, and why choosing the key is a design decision with consequences for ordering and parallelism.
- Describe consumer groups, offsets, and rebalances, and reason about what happens to in flight work when a consumer joins or leaves.
- Distinguish at least once, at most once, idempotent producer, and exactly once via transactions, and choose among them for a money moving event.
- Justify the partition count as the hard ceiling on consumer parallelism, and reason about the cost of resizing it.
- Explain retention, replication, and log compaction as three independent controls over durability, history, and state.
- Connect the outbox to Debezium and articulate precisely why dual writes are forbidden in a money system.
Before we dive in
Suppose the order service, on a fill, updates the balance in Postgres and then publishes an event to Kafka. We saw in the PostgreSQL concept why this fails: Kafka does not participate in a database transaction, so a failure between the two operations produces either a phantom event or a lost one. The fix was the transactional outbox, where the event row lives in the same transaction as the balance change. But the outbox only solves the question of whether the event exists. The moment you have solved that, a second, larger question appears: how do you move that event to every service that needs it, reliably, in order, at scale, without each service re reading the database.
That question is what Kafka was built to answer. The naive answer, a message queue, treats each message as something to be delivered once and forgotten. Kafka treats a message differently: it is an immutable record appended to a durable log, and consumers read that log at their own pace, keeping a bookmark of how far they have gotten. The difference sounds small until you realize it changes everything downstream. Because the log is retained, a consumer that crashes can resume from its bookmark rather than losing work. Because the log is append only and ordered within a partition, a consumer can reason about ordering. Because consumers are grouped and assigned partitions, parallelism is explicit and bounded. The mental shift is from delivery to subscription over a committed history.
The append-only log as the center of gravity
A Kafka topic is a named stream of records. Each record is a key, a value, a timestamp, and optional headers. The topic is split into partitions, and each partition is an ordered, append only sequence of records stored on disk. A producer appends to the end of a partition. A consumer reads forward from an offset, a monotonically increasing integer position within the partition. Nothing is ever modified or deleted inside the active window of the log, except by retention or compaction policies that act on whole segments.
This single design choice, an immutable ordered log, is why Kafka behaves so differently from a traditional broker. Writes are sequential appends, which are fast on disk and easy to replicate. Reads are sequential scans from an offset, which lets a consumer replay history exactly. Durability comes from replicating each partition to a set of brokers, and ordering comes from the partition being a single sequence. The cost of this simplicity is that any global property, like total ordering across a topic, is something you must engineer, because Kafka offers ordering only within one partition.
Topics, partitions, and the ordering guarantee
The most important invariant to internalize is the scope of ordering. Within a single partition, records appear in the order they were appended. Across partitions, there is no ordering guarantee. If you need two events to be observed in a specific order, they must land in the same partition. If you need many independent streams processed in parallel, you spread them across partitions and accept that events in different partitions may interleave.
The implication for a brokerage is concrete. All events for one customer, or one order, or one instrument, must share a partition if downstream consumers need to process them in order. Events for different customers can be freely parallelized. The mechanism that assigns events to partitions is the message key.
A worked example makes the tradeoff visible. Consider a customer events topic keyed by customer id. Every event for one customer, a deposit, an order, a dividend, lands in one partition and is consumed in order, which is exactly what a portfolio service needs to reconstruct that customer’s state. Now consider a market data topic keyed by instrument id. Every quote for one instrument orders within a partition, which is what a yield calculator needs. The two topics use different keys because they need ordering over different entities, and neither key implies anything about the other. The mistake is to key a topic by something broader than the entity that needs ordering, which overconstrains parallelism, or narrower, which breaks the ordering the consumer depends on. The key must be exactly the entity whose order matters, no more and no less.
The key decides the partition
When a producer sends a record with a non null key, Kafka hashes the key and maps the hash to a partition. The default strategy is a deterministic hash, so the same key always lands in the same partition. This is how Kafka gives you ordered processing per entity: choose the entity id as the key, and every event for that entity flows through one partition in order. A null key, by contrast, is assigned round robin or sticky, which spreads load but gives no ordering.
Choosing the key is therefore one of the most consequential design decisions in a Kafka system, and the engineering blog’s outbox material makes the choice explicit. The outbox row carries a topic name and a topic key. The EventRouter, the Debezium component that reshapes outbox changes into Kafka records, uses the topic key as the Kafka record key. For a customer event the key is the customer id, so all of that customer’s events order within a partition. For a market data event the key is the instrument id, often an ISIN. The key is not an afterthought; it is the lever that trades ordering guarantees against parallelism, and it must be chosen by the semantics the consumer needs.
Producers, consumers, and consumer groups
A producer appends records. A consumer reads them. The interesting structure is the consumer group, the mechanism that lets a set of consumers cooperate to process a topic in parallel while each record is handled by exactly one consumer in the group.
Each partition of a topic is assigned to exactly one consumer within a consumer group. If a topic has three partitions and the group has three consumers, each consumer owns one partition and processes it independently. If the group has more consumers than partitions, the extra consumers sit idle, because a partition cannot be split. If the group has fewer consumers than partitions, each consumer owns several. This assignment is the unit of parallelism, and it is the reason partition count is the ceiling on throughput within a group.
A second consumer group reading the same topic gets its own independent offsets. This is how one Kafka topic feeds many downstream systems without coupling them: the notification service, the analytics pipeline, and the audit logger each have their own group and their own bookmark into the same log. One topic, many projections, no interference.
Offsets and the rebalance
Each consumer commits its offset, the position it has reached in its partition, back to Kafka, traditionally stored in an internal offsets topic. Committing an offset is the consumer’s promise that it has processed everything up to that point. If the consumer crashes, the partition is reassigned to another consumer in the group, and that consumer resumes from the last committed offset. This rebalance is how Kafka handles consumer failure transparently, but it is also where subtle correctness bugs hide.
The rebalance is triggered whenever the membership of a consumer group changes: a consumer joins, leaves, or crashes. During a rebalance, partition ownership is shuffled, and consumers may briefly stop processing. The choice of when to commit offsets determines the delivery semantics. Commit before processing and you risk losing work on a crash, because the offset advances past records the consumer never finished, which is at most once. Commit after processing and you risk duplicating work on a crash, because the offset stays behind records already processed, which is at least once. For a money moving event you almost always choose at least once and make the consumer idempotent, because duplicates are recoverable and losses are not.
The rebalance has a stop the world phase that a senior engineer must account for. During a cooperative rebalance the protocol revokes only the partitions that must move, so the rest keep processing, which minimizes the pause. During an eager rebalance every consumer gives up all its partitions and waits for a fresh assignment, which is a fuller pause but simpler and safer when consumers hold state that must be cleanly handed off. The choice of rebalance strategy matters most for stateful consumers, where a pause also means flushing and rebuilding state stores, which is the bridge to the Kafka Streams concept that follows.
A practical hazard is the slow consumer that triggers a session timeout. Each consumer sends heartbeats to the broker to prove it is alive. If a consumer is stuck in a long poll cycle or a slow side effect, it stops heartbeating, the broker declares it dead, and a rebalance starts even though the consumer has not crashed. The rebalance assigns its partitions elsewhere, but the original consumer may wake up and commit stale offsets, causing duplicates or confusion. The defense is to keep poll cycles shorter than the session timeout, to push heavy work out of the consumer thread, and to size the session timeout for the worst case processing latency, not the average.
The two failure modes that follow a slow consumer, false death and offset drift, are worth distinguishing. False death is when a healthy but slow consumer is declared dead by the broker, triggering a needless rebalance that duplicates work across the old and new owners until the old one is finally shut down. Offset drift is when commits lag the actual processing, so a crash reprocesses a window of records. Both are caused by the same root, work that outpaces the consumer’s configured cadence, and both are fixed by the same discipline: separate the consume loop from the side effect, so the poll thread only reads and heartbeats while a worker pool does the heavy work, and commit offsets only after the worker confirms the side effect. A consumer that blocks its poll thread on a slow downstream is a rebalance waiting to happen.
The delivery semantics ladder
Delivery semantics are not a single setting; they are a ladder of guarantees, each bought with more mechanism and more cost. A senior engineer reasons about which rung the workload needs and pays only for that.
At most once means a record is delivered zero or one times. You get it by committing the offset before processing, or by an auto commit that fires on a timer regardless of whether processing finished. It is cheap and fast, and it is acceptable only where losing a record is tolerable, which in a brokerage is almost nowhere.
At least once means a record is delivered one or more times. You get it by committing the offset only after processing has succeeded. On a crash, the consumer reprocesses everything since the last commit. This is the default expectation for financial events, and it places a hard requirement on the consumer: processing must be idempotent, so that reprocessing the same record produces no double effect. A ledger entry must be keyed by a deduplication id so a replay does not post twice.
Exactly once means a record is delivered exactly one time. True exactly once across independent systems is impossible without distributed consensus. Kafka provides it within its own boundaries through two mechanisms that compose: idempotent producers and transactions.
At least once and the poison pill
The at least once model has a failure mode every senior engineer must recognize. Suppose a consumer reads a record, processes it, but processing throws because the payload is malformed or a downstream dependency is down. If the consumer commits the offset to move on, the record is lost. If the consumer refuses to commit and blocks, the partition stalls and no further records are processed. That blocking record is the poison pill, and it is the single most common cause of a stuck Kafka pipeline.
The engineering blog’s outbox material addresses this directly: messages are grouped by key so that independent groups can be retried in isolation, and a poison pill in one group does not block the others. The broader principle is that a consumer must have a strategy for records it cannot process: retry with backoff, route to a dead letter topic, or quarantine. Silently blocking is never acceptable at brokerage scale, because one bad record freezes an entire partition and every well formed record behind it.
A worked ledger write makes the idempotence requirement concrete. A consumer reads an event that says a customer deposited ten euros, and its job is to post that deposit to the customer’s ledger. It commits the offset after the ledger write succeeds. Now suppose the consumer crashes after the ledger write but before the offset commit. On restart it re reads the same event and writes the deposit a second time. Without idempotence, the customer’s balance is now twenty euros too high, a money bug. The fix is that the ledger write carries a deduplication id, derived from the source topic, partition, and offset, and the ledger enforces uniqueness on that id. The second write collides on the id and is rejected, so the customer is credited once. This is what at least once plus idempotence buys: the consumer may process a record twice, but the effect happens once, because the side effect is keyed by where the record came from.
Idempotent producers and transactions
Kafka’s idempotent producer solves a narrower but real problem. A producer that retries a send after a network timeout might produce the record twice, because the original send may have succeeded but the acknowledgment was lost. The idempotent producer attaches a producer id and a sequence number to each record, and the broker deduplicates based on them, so a retried send is collapsed to a single write. This gives exactly once delivery from one producer to one partition, which removes the most common source of producer side duplicates.
Transactions extend this across partitions and across the consume process produce cycle. A consumer that reads from an input topic, transforms, and writes to an output topic wants to commit its input offset and its output writes atomically. If it crashes mid way, you want either both the outputs and the offset commit to happen, or neither, otherwise you get duplicates or losses. Kafka transactions let a producer begin a transaction, write to multiple partitions, send the offset commits for the input partitions, and commit or abort atomically. Consumers that read with the read committed isolation level see only records from committed transactions.
The trap is to assume exactly once is free or universally available. Kafka transactions give exactly once within Kafka, across partitions and the consume produce cycle. They do not give exactly once delivery to an external system like Postgres or Redis. Once you leave Kafka, you are back to at least once and must make the side effect idempotent. A candidate who claims end to end exactly once across Kafka and a database has overreached.
Partition count is the parallelism ceiling
The partition count of a topic is the maximum number of consumers that can process it in parallel within one consumer group, because each partition is assigned to exactly one consumer. If you have six partitions, the most parallelism you can buy is six consumers; a seventh sits idle. This makes the partition count a decision with long consequences, because increasing it later is not free.
When you add partitions, you change the key to partition mapping. A key that hashed to partition three may now hash to partition five, which breaks any ordering guarantee consumers relied on. Existing data stays in its old partition, so the same key now spans two partitions, and ordering across them is undefined. For this reason partition count is usually chosen up front with headroom, based on the peak throughput you expect and the per partition rate a single consumer can sustain.
You cannot cheaply scale Kafka throughput by adding consumers if you have too few partitions, because extra consumers go idle. You cannot safely add partitions later if consumers depend on key ordering, because the key to partition map shifts. Partition count is the one structural decision you must get right early, sized for peak parallelism with margin to spare.
There is also a per partition cost. Each partition is its own set of files, its own indexes, its own replication traffic, and its own metadata the broker tracks. Thousands of partitions across thousands of topics stress broker memory and slow controller operations. The senior instinct is to choose enough partitions for parallelism but no more, and to partition by topic boundaries rather than fragmenting a single logical stream.
Retention, replication, and compaction
Three independent controls govern how long data lives and how safe it is. Retention decides how long Kafka keeps records before deleting them, by time, by size, or both. A time based retention of seven days means a consumer that falls behind by more than seven days loses old data, which forces a reset. Replication decides how many copies of each partition exist across brokers; a replication factor of three tolerates one broker failure without data loss. An under replicated partition, one with fewer than the desired copies, is an alert that durability is degraded.
Compaction is the third control and the one that turns a topic into a source of state rather than a stream of events. A compacted topic keeps only the latest value for each key, deleting older records with the same key. This lets you model a changelog: each record updates the value for its key, and compaction retains the current state. Compacted topics back Kafka Streams state stores and any use case where you want the latest per key without an unbounded history. Retention and compaction can combine: compact by key, and also drop records older than a window.
These three are independent. Retention bounds history by time. Replication bounds durability by copies. Compaction bounds history by key recency. An interviewer will check that you do not conflate them, especially that you understand compaction does not delete the latest value for a key and that replication factor is about fault tolerance, not performance.
The outbox and the dual write prohibition
The connection between Kafka and the outbox is the spine of the whole architecture. The outbox writes the event row in the same database transaction as the business change, so the event’s existence is atomic with the state change. Debezium tails the Postgres WAL via logical replication and emits each outbox row as a Kafka record, using the EventRouter to set the Kafka topic from the row’s topic name and the Kafka key from the row’s topic key.
The reason this design exists, and the reason the dual write is forbidden, is the transaction boundary. A balance update and a Kafka publish cannot share one transaction, because Kafka does not participate in a PostgreSQL transaction. Wrapping both risks phantom events, when the publish succeeds but the commit rolls back, and long running transactions, when the publish blocks the transaction while it waits for a Kafka acknowledgment. The outbox removes the cross system transaction entirely: the application commits only to Postgres, and the Kafka side is fed asynchronously from the WAL. Either both the balance change and the event row commit, or neither does. The stream may lag, but it never diverges from the truth.
Notice what the design earns. The database is the single source of truth. Kafka is a tail of the log, which means it can be rebuilt from the database if needed. Consumers are decoupled, each with its own group and its own pace. And the publish path adds near zero load to the database, because Debezium reads the WAL rather than polling the table.
What breaks at scale: operating the log
A Kafka cluster at brokerage volume has a small set of failure modes, and each maps to a metric a senior engineer watches.
The first is consumer lag. A consumer group falls behind when its processing rate is below the produce rate, so the distance between the log end and the committed offset grows. Lag is not always an error; a bursty market can produce a spike that a consumer catches up during a lull. The danger is sustained lag, which means the consumer is undersized for its partition share, or its side effects are too slow. The fix is to parallelize by adding partitions and consumers up to the partition ceiling, to push slow side effects out of the consumer thread, or to shed load deliberately with a dead letter strategy rather than blocking.
The second is under replicated partitions. A healthy partition has its full replication factor of in sync replicas. When a broker slows or fails, some partitions drop below their factor, which degrades durability: another failure before repair can lose data. Under replicated partition count is the single most important broker health metric, and it should be zero in steady state.
The third is the hot partition. If the key distribution is skewed, one partition receives far more traffic than the others, and the consumer owning it becomes the bottleneck while others sit idle. A single instrument that trades in a frenzy, keyed by its ISIN, can create this. The remedies are uncomfortable: you cannot move one key’s traffic without changing the keying scheme, so the usual response is to ensure the consumer for the hot partition is not also doing expensive side effects, or to introduce a sub key that splits the hot entity across partitions at the cost of cross partition ordering.
The message key simultaneously decides ordering scope and load distribution, and the two pull against each other. A coarse key gives strong ordering but risks a hot partition; a fine key spreads load but breaks per entity ordering. Choosing the key is choosing which guarantee you will weaken under skew, and a senior engineer makes that choice deliberately, knowing the hot partition is the consequence of preferring ordering.
The fourth is retention pressure. A topic with time based retention holds data for its window regardless of whether anyone consumes it, so disk usage is governed by produce rate times retention. A topic sized for seven days at peak market volume must have the disk headroom for that, or retention pressure causes unexpected deletion of data a slow consumer still needed. Sizing topics for peak, not average, is the discipline, and compaction, where the workload allows it, bounds disk use to the number of distinct keys rather than the time window.
Defending the design in an interview
The interview questions on Kafka probe whether you understand it as a log, not a queue, and whether you can reason about ordering, parallelism, and delivery guarantees as design levers rather than configuration knobs.
Why partition at all, and what does the key decide? Partitioning buys parallelism and scalability; the key decides ordering scope, because ordering holds within a partition. Why is partition count a ceiling and hard to change? Because each partition goes to one consumer, and adding partitions shifts the key map and breaks ordering for existing keys. What is the difference between at least once and exactly once in Kafka? At least once is the default financial model, made safe by idempotent consumers; exactly once within Kafka needs idempotent producers and transactions, and does not extend to external systems. Why is the outbox plus Debezium better than dual writes? Because it removes the cross system transaction, makes the database the source of truth, and feeds Kafka from the WAL with near zero extra load.
Common mistakes candidates make
- Claiming Kafka guarantees global ordering. Ordering holds only within a partition; cross partition ordering is never guaranteed, and any design that assumes it is wrong.
- Ignoring the key. Treating the message key as optional when consumers need per entity ordering, then being surprised that events interleave.
- Treating partition count as elastic. Adding consumers beyond the partition count does nothing, and adding partitions later breaks key ordering.
- Confusing delivery semantics. Stating exactly once is the default, or that it extends to Postgres and Redis. At least once plus idempotent consumers is the financial reality.
- Letting a poison pill block a partition. A consumer with no dead letter strategy freezes an entire partition behind one bad record.
- Equating replication factor with performance. Replication is for durability and fault tolerance, not throughput.
- Forgetting that offsets are the consumer’s responsibility. Auto commit before processing gives at most once and silent loss, which is unacceptable for money.
- Believing the consume transform produce cycle is safe without transactions. Without transactions, a crash mid cycle duplicates or loses records.
- Ignoring consumer lag until it becomes an outage. Sustained lag means the consumer is undersized; sizing for peak produce rate, not average, is the discipline.
- Keying too coarsely and creating a hot partition. A key wider than the entity that needs ordering overconstrains parallelism and can funnel a burst onto one consumer.
What an interviewer asks next
A strong candidate expects the follow up, not just the first answer. After the partition ceiling, expect: how do you size partitions up front, given you cannot easily resize? The answer is to estimate peak produce rate per partition and peak consume rate per consumer, then choose enough partitions that one consumer can keep up at peak with margin, typically sizing for two to three years of growth. After the delivery semantics ladder, expect: what does idempotent actually mean for a ledger write? The answer is that the write carries a deduplication id, often the topic, partition, and offset of the source record, so a replay is detected and collapsed rather than applied twice. After the outbox discussion, expect: what happens if Debezium falls behind? The answer is that the WAL grows via the replication slot, and the mitigation is monitoring slot lag plus a controlled catch up, never letting it reach disk full.
Mastery Questions
Question
You need every event for one customer processed in order, but you also want high throughput. How do you reconcile these with Kafka's design?
Answer
Use the customer id as the message key. Kafka hashes the key to a fixed partition, so all events for one customer land in one partition and are delivered to one consumer in order. Throughput comes from spreading many customers, each a different key, across many partitions. Ordering is per key, not global, which is exactly what a brokerage needs: ordered within an entity, parallel across entities.
Sources & evidence8 claims · 2 cited
Eight claims: five sourced from Trade Republic Debezium and outbox material (dual-write prohibition, EventRouter key, BYTEA payload, polling versus WAL), three stable-common-knowledge on partitions, consumer groups, and delivery semantics. The partition-count-is-a-ceiling claim is sourced from the outbox post.
- A Kafka topic is a partitioned append-only log; ordering holds within a partition but not across partitions, and the partition is the unit of parallelism, ordering, and state ownership.stable common knowledge
- The outbox EventRouter uses the row's topic_key as the Kafka record key, which decides the partition and therefore the ordering scope; the key must be the entity whose events must stay ordered.verified
- A balance update and a Kafka publish cannot share one transaction because Kafka does not participate in PostgreSQL transactions; the outbox writes the event row in the same transaction as the business change so both succeed or both fail.verified
- A polling-based outbox adds redundant polling, extra database load, a latency-vs-load tradeoff, and operational complexity; Debezium reading the WAL is far more scalable than polling.verified
- Within a consumer group each partition is assigned to exactly one consumer, so partition count is the hard ceiling on parallelism; a second group reads the same topic at independent offsets to feed multiple downstream systems.stable common knowledge
- Committing the offset only after processing gives at-least-once delivery, which is the financial default and requires idempotent consumers; idempotent producers and transactions give exactly-once within Kafka but not across external systems.stable common knowledge
- Partition count is a structural decision that is hard to change later because adding partitions shifts the key-to-partition mapping and breaks ordering for keys already in flight.verified
- The outbox payload uses BYTEA so Debezium passes bytes to Kafka without database-side parsing, making the format agnostic and avoiding Debezium-level poison pills.verified
Cited sources
- Streaming Outbox Events from Postgres to Kafka with Debezium · Trade Republic Engineering (Andrei Sukharev)
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)