On this path · Distributed Primitives 2. Change Data Capture and Event Reliability
Change Data Capture and Event Reliability
WAL logical replication, Debezium, Avro and Schema Registry, and the dual-write problem CDC solves.
Learning outcomes
An outbox table gives you an atomic place to record events, but it does not deliver them. Something has to read the unpublished rows and push them into Kafka. There are two ways to build that something: a polling loop that queries the table, and change data capture that reads the database’s own write-ahead log. The choice is where you pay the cost, and Trade Republic’s migration from polling to Debezium is the cleanest published case study of the trade.
After studying this page, you can:
- Explain the dual-write problem and why the outbox is the local-consistency fix for it.
- Contrast a polling publisher against CDC from the WAL, and name what each costs in latency, database load, and operational complexity.
- Describe how Debezium uses PostgreSQL logical replication to capture committed changes with low latency and near-zero table-query overhead.
- Justify application-side Avro serialization, BYTEA payloads, and explicit schema registration over Debezium-side conversion.
- Explain the EventRouter pattern: a topic-name column drives routing, and the topic key becomes the Kafka partition key.
- Diagnose poison pills, schema drift, and the failure modes unique to each delivery approach.
Before we dive in
The previous concept established why the outbox exists: two independent systems, the database and Kafka, cannot share a transaction, so a business change and its event can diverge if a crash falls between them. The outbox fixes the atomicity half by writing both rows in one transaction. What it leaves open is the delivery half. The event row is now safely in the database, but it still has to reach Kafka, and the mechanism that moves it is a separate engineering problem with its own failure modes.
The delivery problem is harder than it looks because the outbox is a high-churn table. Rows pour in from every trade, every balance change, every order event, and they have to leave again once published. Whatever drains the table must keep up with the insert rate, it must preserve order, it must not starve the database of resources, and it must handle failures without losing or duplicating events. There are two architectures for this drainer, and they sit at opposite ends of a spectrum. The polling publisher asks the database a question over and over. Change data capture listens to what the database already does internally. Trade Republic ran both and wrote about why it moved from the first to the second.
The two ways to drain an outbox
A polling publisher is an application loop that periodically queries the outbox table for unpublished rows. It runs in a service, opens a connection, asks for the next batch in order, publishes to Kafka, and marks the rows. The database is passive: it answers queries when asked. The polling interval sets a floor on latency. If the loop runs every second, no event can be delivered faster than that second, regardless of how fresh it is.
Change data capture, or CDC, inverts the relationship. Instead of the application asking the database for new rows, the database tells the application about changes as they happen. PostgreSQL, like every transactional database, writes every committed change to a write-ahead log (the WAL) for crash recovery before it touches the main table. The WAL is the source of truth for what changed and in what order. CDC reads that log, decodes the row changes, and streams them out. The database is not queried for the outbox at all; the changes are observed as a side effect of the recovery mechanism the database already runs.
The difference is not cosmetic. Polling adds load to the database proportional to how often you ask, and it bounds latency by the asking interval. CDC adds near zero query load, because it reads a log the database maintains anyway, and it delivers changes the moment they commit. The cost of CDC moves from the database to the operational complexity of running a CDC pipeline.
Why polling is expensive
A polling publisher seems simple until you measure what it does to the database. Every poll is a query, and the query touches the outbox table. Even with a good index, each poll does work: it seeks the index, reads the working set, and returns rows. When the table is mostly empty because the publisher keeps up, most polls return nothing, but they still pay the index seek and the visibility check. When the table is busy, the polls compete with the inserts for buffer cache and CPU.
The latency versus load tradeoff is the core tension. To lower latency you poll more often, but each poll is a query, so more frequent polling means more database load. To reduce load you poll less often, but then events sit in the table longer before they are delivered, and downstream systems see stale data. You cannot win both axes with polling. You pick a latency target and accept the query load it implies, or you pick a load budget and accept the latency it imposes.
Polling cannot deliver an event faster than the poll interval, and it cannot lower that interval without adding query load. A one-second poll means one-second best-case latency forever, paid for by one query per second whether or not there is work. CDC breaks this coupling because delivery is event-driven, not timer-driven.
There is a second, subtler cost. A polling publisher owns the publish transaction. It opens a transaction, reads, publishes, marks, and commits. That transaction is in the application’s control, which means the application owns the failure handling, the marking logic, the retry policy, and the transaction length. As the outbox concept showed, getting any of that wrong (long transactions blocking vacuum, poison pills stalling the batch, marking races) directly harms the database. The polling publisher is application code running inside the database’s most sensitive loops.
Change data capture from the WAL
CDC removes the publisher’s query loop by reading from a place the database already maintains: the write-ahead log. Every transactional database writes a WAL. Before PostgreSQL modifies any data page in memory, it writes a record of the intended change to the WAL and flushes it to disk. This is what makes the database crash-safe: on recovery, it replays the WAL to reconstruct committed changes that had not yet reached the main data files. The WAL is therefore a complete, ordered, durable record of every committed change.
PostgreSQL offers logical replication, a mode that decodes the WAL’s physical change records into a logical form: row insertions, updates, and deletions keyed by table and primary key. Logical replication is what lets CDC read committed changes without ever querying the tables. A logical replication slot tracks how far a consumer has read, so the database knows which WAL segments it can safely discard and which it must retain until the consumer catches up.
The properties that follow from reading the WAL are exactly what polling cannot offer. Latency is low because the event appears the instant the transaction commits, not on the next poll tick. Database load from the outbox query is near zero because no table is scanned; the WAL is read sequentially, which is cheap, and it is read once rather than polled repeatedly. Order is preserved because the WAL is itself strictly ordered by commit. The tradeoff is operational: you now run a logical replication slot, a decoding process, and a connector, and you must monitor the slot so it does not fall behind and force the database to retain unbounded WAL.
How Debezium streams committed changes
Debezium is the open-source CDC connector that Trade Republic adopted. It runs as a Kafka Connect connector, attaches to PostgreSQL via a logical replication slot, decodes the WAL, and emits a change event to a Kafka topic for every committed row change. For an outbox table, the interesting row changes are inserts: every business transaction that commits inserts an event row, and that insert appears in the WAL, which Debezium captures and streams.
The key advantage over polling is that Debezium captures only committed changes, which eliminates a whole class of phantom-event bugs. A polling publisher that reads inside its own transaction might see rows that are later rolled back, then publish them, then discover the rollback. Because Debezium reads the WAL, and only committed changes reach the WAL in their final form, it never sees an uncommitted or rolled-back row. The change event is the durable truth of what the database committed.
Logical replication decodes the WAL, and the WAL records committed transactions. Debezium therefore streams only changes that are durable, which means the downstream consumer never receives an event for a business change that was rolled back. This is a stronger guarantee than a polling publisher can easily give.
Scalability is the second advantage. A polling publisher is one application competing for database resources, and scaling it means running more publisher instances, each querying the table, which adds load. Debezium’s load on the database is the cost of decoding the WAL, which is largely fixed by the write rate and does not grow with the number of consumers. Once the change is in Kafka, any number of downstream services read it independently, and none of them touches the database.
Avro serialization on the application side
The outbox payload has to reach Kafka in a format consumers can parse. There are two places to serialize it: in Debezium, after it reads the WAL, or in the application, before it inserts the outbox row. Trade Republic chose application-side serialization, and the reasoning is worth understanding because it is a classic fail-fast versus fail-late tradeoff.
If Debezium serializes, the application inserts a structured row and Debezium converts it to Avro on the way out. The problem is that schema mismatches and serialization errors happen in Debezium, at the CDC layer, far from the business code that produced the data. A bad row discovered by Debezium is a poison pill at the connector level, which can stall the whole replication slot and back up WAL retention on the database. The failure is late, indirect, and hard to attribute.
Serialization on the application side means the business code serializes the event to Avro bytes and inserts those bytes into the outbox as a BYTEA column before the transaction commits. If the serialization fails, the business transaction fails immediately, at the point where the data is produced, where the error is actionable and attributable. The row in the outbox is already-serialized bytes, and Debezium simply passes the BYTEA through to Kafka without parsing or converting it.
Schema registration is the other half of the fail-fast argument. When the application serializes, it registers schemas explicitly with a schema registry at deploy time, not at runtime. A producer deploying a new schema version registers it before it starts writing, so consumers know the schema exists. With Debezium-side serialization, schema registration can happen at runtime during decoding, which means a new schema is discovered only when the first matching row flows through, which is unpredictable and can surprise consumers.
Storing the payload as BYTEA means the outbox holds opaque bytes. Debezium does not parse or convert them; it emits the bytes as the Kafka message value. This keeps the payload format agnostic to the database and the connector, so changing the serialization format is an application concern, not a CDC reconfiguration.
Routing, keys, and the EventRouter
An outbox feeds many topics. Trade events go to one topic, balance changes to another, order lifecycle events to a third. A naive CDC setup sends every outbox row to a single topic and lets consumers filter, which wastes bandwidth and couples consumers to a shared firehose. The EventRouter is the Debezium pattern that routes each row to its correct topic.
The outbox table carries a topic_name column that records which topic the event belongs to, and a key column that becomes the Kafka message key. The Debezium EventRouter single message transform reads the topic_name to decide the destination topic and uses the key column as the Kafka partition key. Routing is therefore data-driven: the application decides the topic and key when it inserts the row, and the connector honors that decision.
The Kafka key deserves attention because it controls partitioning. Kafka guarantees that messages with the same key land on the same partition and are read in order by a single consumer within a consumer group. For trade events, the key is the account identifier, so all events for one account land in order on one partition. This ordering guarantee is what lets a downstream portfolio service process a customer’s events sequentially without reordering bugs.
The contrast with a polling publisher is instructive. A polling publisher hardcodes its topic logic in application code: it reads a row, inspects a field, and publishes to the right topic. That logic lives in the publisher, which means changing routing means redeploying the publisher. With CDC and the EventRouter, routing is a property of the data, declared at insert time, and the connector enforces it. The application that produces the event owns the routing decision, and the delivery infrastructure simply honors it.
Delivery semantics and the myth of exactly once
A question interviewers return to is whether the pipeline delivers exactly once. The honest answer is that the CDC layer delivers at-least-once, and exactly-once is an end-to-end property the consumer must construct. Debezium reads the WAL and writes to Kafka, committing its Kafka producer offsets periodically. If the connector process dies after emitting a batch to Kafka but before recording how far it read, the replacement connector starts from the last recorded offset and re-emits the in-flight batch. The same change event appears in Kafka twice.
This duplicate is unavoidable at the CDC layer without a distributed transaction across the WAL read, the Kafka write, and the offset commit, which no system actually provides cheaply. The consequence is that every downstream consumer must treat events as possibly duplicated and deduplicate against a stable event identifier, which the outbox row’s sequence identifier provides. A consumer that applies a balance event twice without deduplication credits or debits the same amount twice, which is a money bug. Idempotent consumption, keyed on the event identifier, is the contract that turns at-least-once delivery into effectively-once application state.
What breaks at scale
At low volume a single Debezium connector reading one replication slot keeps up easily, and the operational surface is small. As the outbox churn grows, three things break in sequence, and each has a known response.
The first is WAL retention pressure. The replication slot pins every WAL segment the consumer has not yet confirmed. As the insert rate climbs, the WAL grows faster, and a connector that is even slightly slow, perhaps because Kafka is throttling or the connect cluster is under-provisioned, lets WAL accumulate. The defense is not a bigger disk but a faster consumer: scale the connect workers, tune the batch size the connector emits, and alert on slot lag measured in bytes or seconds behind. A slot that grows without bound is a disk-exhaustion incident waiting to happen, and the right action is to fix the consumer throughput, not to widen the disk.
The second is connector throughput. A single connector task processes one replication slot sequentially, because the WAL is a single ordered stream. You cannot parallelize the read of one slot. When one slot cannot keep up, the response is logical sharding: split the source into multiple logical databases or schemas, each with its own outbox and its own replication slot, and run a connector per slot. This trades one ordered stream for several, each handling a slice of the traffic, at the cost of losing global ordering across slices (which you rarely need, since ordering matters per account, not globally).
The third is schema evolution. The outbox payload is Avro, and Avro schemas evolve under rules: fields can be added with defaults for backward compatibility, types can be promoted under widening rules. A producer that deploys a new schema version must register it before writing, and consumers must tolerate both the old and new versions during the rollout window. Application-side serialization makes this explicit: the producer registers at deploy time, and a registration failure stops the deploy, so a schema that breaks compatibility never reaches production. With connector-side serialization, a breaking schema can flow through undetected until a consumer fails to decode, which is a late, broad outage.
Treat schema registration as part of the deploy, not the runtime. A producer that registers its schema on startup, before accepting traffic, fails the deploy if the schema is incompatible, keeping breaking changes out of production. Consumers read the registry for the writer schema and evolve under Avro compatibility rules, so a rolling rollout sees a mix of versions without decoding failures.
Common mistakes candidates make
The first mistake is confusing CDC with a cure for the dual-write problem. CDC does not make the database and Kafka atomic; the outbox does. CDC is the delivery mechanism for the outbox. A candidate who reaches for Debezium to solve consistency has the dependency backwards: you need the outbox for atomicity, and then you choose CDC or polling to drain it.
The second mistake is assuming CDC delivers exactly once. Debezium delivers at-least-once. If the connector crashes after reading the WAL but before committing its Kafka offset, it re-reads and re-emits on restart. Downstream consumers must be idempotent, deduplicating by a stable event identifier. Exactly-once semantics across Kafka and the database are an end-to-end property achieved through transactional sinks and idempotent consumers, not a property of the CDC layer.
The third mistake is ignoring WAL retention. A logical replication slot tells PostgreSQL how far the consumer has read. If the consumer falls behind or stops, PostgreSQL cannot discard the WAL the slot still needs, so WAL files accumulate on disk until it fills up. A CDC deployment without slot monitoring is a time bomb: a stuck connector silently grows WAL until the database runs out of disk and stalls. Monitoring replication slot lag is a first-class operational concern.
The fourth mistake is choosing Debezium-side serialization for convenience and inheriting its failure modes. A schema mismatch discovered at decode time is a connector-level poison pill that can block the replication slot, which backs up the database. Application-side serialization fails fast, at the source, before the row is ever written, keeping the bad data out of the pipeline entirely.
The fifth mistake is forgetting that ordering is per-partition, not global. CDC preserves the WAL’s commit order, but once events land in Kafka, ordering holds only within a partition keyed by the same key. A candidate who assumes global ordering across topics or across accounts will design a consumer that races on unrelated events. The key must be the entity whose order you care about, typically the account.
Defending the design
An interviewer will probe why you chose CDC over polling, and the defense has three parts. First, latency: CDC delivers committed changes with no poll interval floor, because delivery is driven by the commit, not by a timer. Second, database load: CDC reads the WAL sequentially and queries no table, so the outbox imposes near-zero query overhead on the database, where polling adds load proportional to the poll rate. Third, scalability: any number of downstream consumers read from Kafka without touching the database, and adding a consumer does not add database load.
The follow-up question is about the cost, and the honest answer is operational. You run a logical replication slot, which means you monitor slot lag, you manage WAL retention, and you run the Debezium connector as a piece of infrastructure with its own failure modes and upgrades. CDC trades query load for operational complexity, and for a high-throughput brokerage where the outbox churns constantly, the trade is worth it because the database is the most expensive and most shared resource.
The last probe is about poison pills and schema evolution. The defense is application-side Avro with explicit schema registration, so serialization failures are fail-fast at the source, schemas are registered at deploy time, and the BYTEA payload keeps the connector agnostic to the format. When a bad row does occur, it is an application bug to fix in the producer, not a connector stall to fight at the CDC layer.
A deeper probe asks when you would choose polling after all. The honest answer is at low volume, where the operational cost of running a CDC pipeline outweighs the query-load saving. A service with a few hundred events a day can poll every few seconds with negligible database impact and no replication slot to monitor, and the simplicity is worth more than the latency win. CDC earns its complexity when the outbox churns constantly and the polling query load and latency floor become the bottleneck, which is the brokerage’s situation. The senior judgement is not that CDC is always better, but that it is better once polling’s cost crosses a threshold that a measurement, not an intuition, reveals.
Mastery Questions
Question
Why can a balance update and a Kafka publish not share one transaction, and how does the outbox address this?
Answer
Kafka does not participate in PostgreSQL transactions, so the publish is a side effect the database cannot roll back; wrapping both risks phantom events (rollback after publish) and long-running transactions (holding the tx open over a network call). The outbox writes the business row and an event row in one database transaction, so they commit atomically, then a separate mechanism drains the event row to Kafka.
Sources & evidence7 claims · 1 cited
Dual-write, polling-vs-CDC, Debezium, Avro, BYTEA, and EventRouter claims backed by Trade Republic Debezium source; WAL-retention hazard and slot-lag monitoring is internal-reasoning.
- A balance update and a Kafka publish cannot share one transaction because Kafka does not participate in PostgreSQL transactions; the publish is a side effect the database cannot roll back, and wrapping both risks phantom events and long-running transactions.verified
- A polling outbox adds redundant polling, extra database load, a latency floor set by the poll interval, and operational complexity, versus Debezium which streams the PostgreSQL WAL via logical replication capturing committed changes with low latency and near-zero extra table-query overhead.verified
- CDC via logical replication decodes the WAL, and only committed transactions reach the WAL in their final form, so Debezium emits only changes the database committed and never produces a phantom event for a rolled-back transaction.verified
- Application-side Avro serialization before insertion fails fast on schema mismatch at the source and avoids Debezium-level poison pills; schema registration is managed explicitly at deploy time, not discovered at runtime during decoding.verified
- The outbox payload uses BYTEA so Debezium passes bytes to Kafka without database-side parsing, making the format agnostic and keeping the connector free of serialization failures.verified
- The Debezium EventRouter routes outbox rows by reading a topic_name column to choose the destination topic and using a key column as the Kafka partition key, so routing is data-driven and decided at insert time by the producer.verified
- A logical replication slot tells PostgreSQL how far the CDC consumer has read; if the consumer falls behind, PostgreSQL cannot discard the needed WAL and the disk fills, so monitoring slot lag is a first-class operational concern.internal reasoning
Cited sources
- Streaming Outbox Events from Postgres to Kafka with Debezium · Trade Republic Engineering (Andrei Sukharev)