On this path · System Design 5. Design Event-Driven Horizontally Scalable Ingestion
Design Event-Driven Horizontally Scalable Ingestion
Countering the monolithic-ingestion failure mode with partitioned batch workers on Kubernetes.
Learning outcomes
The documented candidate failure at Trade Republic is building a monolithic ingestion service instead of a horizontally scalable one. The design task is to build event-driven ingestion that scales out rather than up: partitioned Kafka topics feeding partitioned workers on Kubernetes, with deduplication and backpressure built in. This page teaches you to design that system and, just as important, to articulate why the monolithic design fails and how the partitioned design escapes each of its failure modes. The deeper partitioning mechanics are taught in Partitioning and Consistent Hashing; this page owns the ingestion architecture.
After studying this page, you can:
- Name the failure modes of a monolithic ingestion service and show how each is structural, not a tuning problem.
- Design a partitioned ingestion pipeline: Kafka partitions keyed by a dedup-friendly key, a consumer group of partitioned workers on Kubernetes, and per-partition state.
- Choose a deduplication strategy, including the UUID-partitioned dedup table, and justify its tradeoffs.
- Design backpressure that degrades gracefully rather than collapsing under burst.
- Reason about exactly-once versus at-least-once with idempotent collapse, and why the latter is the engineering reality.
- Anticipate the Staff-engineer probes on hot partitions, rebalancing, and the operational cost of horizontal scaling.
Before we dive in
Ingestion is the boundary where the outside world meets the system. External feeds, partner systems, and internal producers push events in: market-data quotes, reference-data updates, customer actions from partner integrations, and batch files from data providers. The ingestion layer’s job is to accept these events reliably, deduplicate them, transform them into the internal canonical form, and publish them to downstream topics for the rest of the platform to consume. It is the on-ramp, and its design decides whether the rest of the system scales.
The origin of the partitioned design is the failure of the monolithic one. A monolithic ingestion service accepts events through a single entry point, processes them in one pool of threads or one queue, and writes to downstream stores. It is simple to build, which is why candidates reach for it, and it is wrong at scale for structural reasons. The single processing pool is the bottleneck: when event volume rises, the only way to add capacity is to make the pool bigger, which means a bigger machine, which has a ceiling. The single queue is the single point of failure: if it backs up, the whole ingest path backs up. And the single ordering scope makes parallelism impossible without giving up ordering guarantees the downstream needs.
The senior insight is that horizontal scaling requires partitioning as its primitive, and partitioning must be chosen deliberately. You cannot scale a worker pool horizontally unless each worker can own a disjoint slice of the work, and that requires a key that splits the work evenly and groups related events together. Kafka partitions provide the disjoint slices, and the partition key provides the grouping. The Trade Republic published work on scaling batch jobs on Kubernetes is the production version of this pattern: partitioned workers, each owning a slice, scaling with the slice count. The design task is to apply that pattern to event ingestion and to defend it against the monolithic alternative.
The three categories of ingestion the system must handle
An ingestion system at Trade Republic must handle three distinct event categories, each with its own volume, latency, and correctness profile. Market-data quotes arrive at a high rate, up to a hundred thousand per second, and the ingestion must normalize and publish them with sub-second latency. The tolerance for loss is moderate: a dropped quote is replaced by the next quote, and the downstream analytic will be slightly stale. Reference-data updates from partner integrations arrive at a low rate, a few per minute, but each update must be applied exactly once and must not be lost, because a missed corporate action or instrument-change event can cause a financial error. Customer actions from internal services, such as account updates and preference changes, arrive at a moderate rate and must be applied in order per customer and with idempotency.
The three categories share the same ingestion pipeline design but differ in the partition key choice and the dedup policy. Market-data quotes are partitioned by instrument id, with at-most-once dedup, because a dropped quote is acceptable. Reference-data updates are partitioned by the entity id, with exactly-once dedup enforced by the outbox. Customer actions are partitioned by customer id, with at-least-once dedup and outbox-based idempotency. The design is the same; the configuration is different.
Why the monolith fails for each category
The monolithic ingestion service fails differently for each category, and naming the specific failure mode shows depth. For market-data quotes, the monolith’s single queue creates a bottleneck that caps the quote throughput. Adding capacity means a bigger machine, which has a ceiling, and the cost grows superlinearly with the quote rate because the memory and CPU for the single queue structure grow with the queue depth. The partitioned design distributes the queue across partitions, so each partition handles its share of the quote rate, and adding partitions adds capacity linearly.
For reference-data updates, the monolith’s single ordering scope forces a choice between parallelism and ordering: if the monolith processes updates in parallel, updates for the same entity may be processed out of order, causing a data inconsistency. If the monolith processes updates serially, the throughput is capped at the per-entity update rate times the number of entities, which is wasteful because entities are independent. The partitioned design gives each entity its own ordering scope through the partition key, so updates for the same entity are processed in order and updates for different entities are processed in parallel.
For customer actions, the monolith’s single point of failure means that a crash during processing can lose an in-flight action. The monolith must replay the action from the beginning, which may cause duplicate processing if the action was partially applied. The partitioned design’s per-partition workers and the outbox-based dedup and publish guarantee that an action is either applied exactly once or not at all, even after a worker crash.
Clarify requirements and the scaling trap
Functional requirements. The system ingests events from many producers: external feeds, partner integrations, and internal services. It deduplicates events, because producers often redeliver, and downstream systems must not double-apply. It transforms events into the canonical internal form and publishes them to downstream Kafka topics partitioned for downstream consumption. It tolerates producer surges and slow downstream consumers without losing events.
Non-functional requirements. Throughput must scale horizontally: adding workers adds capacity, up to the partition count. Latency per event must be low in the steady state, and bounded under burst. Correctness is at-least-once delivery with idempotent downstream application, because exactly-once across distributed consumers is not achievable without prohibitive cost. Availability must be high, with no single component whose failure stops ingestion.
The scaling trap is the heart of the prompt, so name it precisely. A monolithic ingestion service scales vertically: to handle more events, you give the one machine more cores or more memory, and eventually you hit the ceiling of a single machine. The bottleneck moves around, from CPU to the single queue to the single downstream connection, but it never disappears, because the architecture has one of everything. A partitioned design scales horizontally: to handle more events, you add partitions and add workers, and the ceiling is the partition count, which you choose in advance. The difference is structural, and no amount of tuning turns a monolith into a horizontally scalable system.
High-level architecture
The pipeline has four layers, and the partitioning runs through all of them as a continuous thread.
The intake layer accepts events from producers and writes them to a Kafka ingest topic. The intake is thin: it validates the envelope, assigns or accepts a producer-supplied id for deduplication, and produces to the topic partitioned by a key chosen for downstream affinity. The intake does no heavy processing, because its job is to get events into the durable topic as fast as possible, where they survive even if the intake itself fails.
The processing layer is a consumer group of workers on Kubernetes, one worker per partition, each owning a disjoint slice of the topic. Each worker reads its partition, deduplicates against a per-partition or shared dedup store, transforms the event to the canonical form, and publishes to the downstream topic. Because each partition has exactly one owner within the consumer group, the worker processes its slice in order and can hold per-key state locally without coordination. The deeper mechanics of partition ownership are taught in Partitioning and Consistent Hashing.
The dedup layer ensures each logical event is applied once, even when producers redeliver. The dedup store is keyed by the event id, typically a producer-supplied UUID, and an event whose id is already present is dropped. The design of the dedup store, including the partitioning strategy that Trade Republic benchmarked, is a deep concern covered below.
The downstream layer is the set of Kafka topics that the rest of the platform consumes: the raw-quotes topic for the streaming pipeline, the reference-data topic for catalog updates, and so on. These topics are partitioned for their own downstream consumers, and the processing worker repartitions as needed when it publishes.
The architecture’s defining property is that there is no single bottleneck. The intake is thin and stateless, so it scales by adding instances behind a load balancer. The processing layer scales by adding workers up to the partition count, and the partition count is chosen for the expected peak. The dedup layer scales by partitioning the dedup table. Each layer fails and scales independently, which is the structural difference from the monolith.
Deep dive: partitioning as the scaling primitive
Partitioning is not an optimization in this design; it is the design. Without it, there is one queue, one worker, and one bottleneck. With it, there are N disjoint slices, N workers, and a throughput that scales with N. The choice of partition key decides whether the partitioning is balanced and whether per-key ordering is preserved, and both matter.
Partition key design patterns
The partition key is the most consequential design decision in the pipeline, and a senior has a taxonomy of patterns to choose from.
The identity key, where the key is the event’s unique identifier, maximizes distribution because the key is unique per event. It provides no ordering guarantee for related events, because two events for the same entity land on different partitions. This is acceptable for market-data quotes, where ordering across quotes for the same instrument is not critical because the downstream handles out-of-order quotes, but it is unacceptable for reference-data updates where per-entity ordering is required.
The entity key, where the key is the entity identifier such as the customer id, the instrument id, or the partner id, groups all events for the same entity on one partition. It preserves per-entity ordering at the cost of distribution: events for a single entity are serialized through one worker. This is the correct pattern for reference-data updates and customer actions, where ordering matters and per-entity throughput is modest.
The composite key, where the key is a combination of the entity identifier and a time bucket, provides better distribution than the entity key while preserving ordering within the time bucket. Events for the same entity within a five-minute window land on the same partition, but events in different windows may land on different partitions. This is useful for high-volume market-data quotes where per-instrument ordering within a short window is sufficient, because out-of-order quotes beyond a few minutes are stale anyway.
The choice of partition key is recorded in the configuration and is part of the topic specification. Changing the partition key requires creating a new topic and migrating consumers, which is a controlled operation. A senior thinks carefully about the partition key before creating the topic, because changing it later is expensive.
Throughput modeling and partition sizing
The partition count is chosen by modeling the expected peak throughput. Each partition can sustain a throughput bounded by the single-threaded processing capacity of the worker. For market-data quotes, a single worker can process tens of thousands of quotes per second before the normalization and dedup logic become the bottleneck. For reference-data updates, the per-worker throughput is lower because the validation and transformation logic is heavier. The partition count is the peak throughput divided by the per-worker throughput, multiplied by a headroom factor of 2x to 3x.
The per-worker throughput is measured by load testing the worker with production-like data. The load test simulates a partition’s worth of events at increasing rates until the worker’s CPU or the dedup database becomes the bottleneck. The throughput at the bottleneck is the per-partition capacity. The partition count is then chosen for the expected peak, with headroom, and is recorded in the topic configuration.
A common error is to choose the partition count based on the number of workers rather than the throughput. If the partition count is lower than the number of workers, the extra workers are idle because the consumer group cannot assign more than one worker per partition. If the partition count is much higher than the number of workers, the workers handle multiple partitions each, which is fine as long as the per-worker throughput is manageable. The partition count should be sized for the peak throughput, and the worker count should be sized to match the partition count at peak load, with fewer workers at off-peak times.
Worker lifecycle and the consumer group protocol
The worker is a Kubernetes pod that runs the Kafka consumer for one or more partitions. The worker’s lifecycle is managed by the Kubernetes deployment controller and the Kafka consumer group protocol.
When a worker starts, it joins the consumer group and is assigned a set of partitions by the group coordinator. The worker initializes its per-partition state: it loads the dedup state for the partition from the dedup table, it initializes the outbox writer, and it starts fetching events. The initialization must be fast, because the partition is not processed during initialization and the consumer lag grows. The dedup state is loaded lazily: the worker reads the dedup table for the partition’s key range on the first event for each key, not for all keys at once. This spreads the initialization cost across the first few seconds of processing.
When a worker is scaled down or redeployed, it receives a signal to stop fetching. The worker commits its offsets for all assigned partitions, closes the outbox writer, and exits. The commit ensures that the next worker to take over the partition starts from the correct offset, with no events lost and no events reprocessed more than necessary. The commit is a synchronous operation with a timeout: if the commit does not complete within the timeout, the worker logs a warning and exits anyway, relying on the next worker’s dedup to handle the reprocessing.
The consumer group protocol handles partition reassignment when a worker joins or leaves. During the reassignment, the partitions are briefly unprocessed, and the consumer lag for those partitions grows. The reassignment duration is typically a few seconds, and the lag growth is bounded by the partition’s event rate during that time. The lag is caught up by the new worker after the reassignment completes. A senior monitors the reassignment duration and the lag spike during reassignments as signals of the consumer group’s health.
The key must balance the load across partitions, so that no partition is a hot spot. For market-data quotes, the instrument id is a natural key: it distributes events across partitions by instrument, and it groups all events for one instrument on one partition, which preserves per-instrument ordering. For partner integration events, the partner plus entity id is natural. The key must also group related events that downstream needs in order, because Kafka guarantees ordering only within a partition. Choosing the key is the act of deciding what must be ordered, and the answer is different per use case.
If one instrument generates a disproportionate share of events, its partition becomes hot and the worker owning it becomes the bottleneck, while other workers sit idle. The mitigation is to detect hot keys at the intake, by sampling, and to split the hot key across sub-partitions with a deterministic suffix, accepting that per-key ordering across the sub-partitions is lost. This is a real scaling limit and the decision to split must be deliberate.
The worker-per-partition model is what gives the design its horizontal scaling property. Kafka’s consumer group protocol assigns each partition to exactly one worker within the group, so adding workers up to the partition count adds parallelism, and each worker processes its partition in order. Beyond the partition count, adding workers does nothing, which is why the partition count is chosen for the peak and increased in advance of growth. The Trade Republic pattern of scaling batch jobs on Kubernetes is the operational version: workers are pods, the partition assignment is the unit of work, and scaling is adding or removing pods up to the partition count.
Rebalancing is the operational cost of the partitioned model. When a worker joins or leaves the group, Kafka rebalances: it reassigns partitions, and during the rebalance, processing pauses for the affected partitions. The mitigation is cooperative rebalancing, which moves only the partitions being reassigned rather than pausing the whole group, and sizing the per-partition state so a new owner recovers quickly. Rebalancing is the moment the design’s correctness is most exposed, because a partition’s offset must be committed and its dedup state must be consistent before the new owner begins.
Outbox table partitioning for ingestion
The outbox in the ingestion pipeline has a different write profile from the matching engine’s outbox. The ingestion outbox receives events from many producers at a high rate, and each outbox row is small, a few hundred bytes. The outbox table must handle the aggregate write rate of all partitions, which can be tens of thousands of rows per second.
The Trade Republic outbox design, documented in the engineering blog and applied here, partitions the outbox by published_at so that the unpublished partition is always small. For ingestion, the unpublished partition holds outbox rows that have not yet been published to the downstream topic. The partition is small, typically a few thousand rows, because the publisher drains it continuously. The published partition holds rows that have been published, and it grows over time. The published partition is archived or pruned after a retention period.
The outbox index must be designed for the high write rate. A partial index on id WHERE published_at IS NULL is the standard pattern, but at the ingestion write rate, the partial index can bloat because the index entries for published rows are not immediately removed. The Trade Republic fix for this bloat is to partition the outbox by published_at and to include the partition key in every UPDATE predicate, so the unpublished partition is the only one scanned during publish. The ingestion pipeline follows the same pattern: the outbox is partitioned, the publisher queries only the unpublished partition, and the published partition is periodically truncated or archived.
The dedup table: hash partitioning in practice
The dedup table is the central correctness mechanism, and its performance at scale determines whether the pipeline can sustain its throughput. The Trade Republic benchmark, published in src_tr_partitioning, compared hash partitioning against list partitioning for a UUID-keyed dedup table and found that hash partitioning outperformed list partitioning on inserts by every metric: higher transactions per second, lower latency, and better scalability.
The practical implementation in the ingestion pipeline follows the benchmark’s result. The dedup table is hash-partitioned into sixteen partitions by default, with the partition key derived from the UUID’s hash modulo the partition count. The primary key on the parent table, (event_id, partition_id), enables fast lookup by event id and allows the database to prune partitions that do not contain the event id. The partition count is configurable and is chosen based on the write rate: a higher write rate requires more partitions to keep the per-partition insert rate below the database’s per-partition throughput ceiling.
A subtle detail the benchmark revealed is that list partitioning on a computed partition key, for example, using the LEFT and LOWER functions to compute a partition key from the UUID, loses the ability to put a primary key on the parent table because partition keys cannot include expressions. List partitioning also requires the partition key computation on every insert, which adds overhead. Hash partitioning uses the built-in hash function, which is implemented in C and is faster, and it preserves the primary key on the parent table, which is required for logical replication. The design choice follows the benchmark: use Postgres hash partitioning for the dedup table.
Dedup table retention and cleanup
The dedup table grows as events arrive and is never naturally pruned. Without a retention policy, it grows unboundedly and eventually degrades in performance and disk usage. The retention policy is based on the maximum redelivery window: the producer is expected not to redeliver an event older than the retention window, which is typically a few hours to a day. After the retention window, the dedup row is no longer needed because any duplicate event would be outside the window and could be applied without causing a duplicate.
The cleanup is partition-level: the dedup table is partitioned by the event id’s hash as described above, and also by time ranges. A cleanup job drops the oldest time-range partition that is outside the retention window. The partition drop is a metadata operation in Postgres and is fast and online. The cleanup runs on a schedule, typically once per hour, and drops any partition whose time range has fully aged out of the retention window.
The retention window is a documented contract with the producer. The producer must not redeliver an event older than the window, or the dedup layer will fail to collapse the duplicate. If a producer violates the contract, the event is applied twice, which the downstream consumer must handle through its own idempotency. The contract is enforced by the ingestion intake, which rejects events with a timestamp older than the window minus a grace period.
Deep dive: deduplication, idempotency, and backpressure
Deduplication is required because producers redeliver. Network retries, producer restarts, and at-least-once delivery guarantees all mean the same logical event can arrive more than once, and downstream systems that apply events, like a ledger or a position update, must not double-apply. The dedup layer collapses redeliveries to one applied event.
The dedup store is keyed by the event id, and the question is how to scale it. A single dedup table in Postgres is the simple answer and works at modest volume, but it becomes the bottleneck as event rate rises, because every event is a lookup and a write against one table. The senior choice, which Trade Republic benchmarked, is to partition the dedup table by the id, so the dedup load spreads across the table’s partitions and the database’s parallelism. The benchmark found that hash partitioning on the id outperformed list partitioning on a computed key, because the hash function is a fast C implementation and hash partitioning preserves the ability to put a primary key on the parent table, which matters for logical replication. The details of that benchmark are in Partitioning and Consistent Hashing.
The dedup decision and the downstream publish must commit together, or a crash between them can lose or duplicate the event. The pattern is the outbox: the worker writes the dedup id and the outbox event in one Postgres transaction, so either both commit or neither does, and Debezium streams the outbox event to the downstream topic. This is the same outbox used across the platform to fix the dual-write, taught in The Outbox Pattern, Deeply. Applying it to ingestion means the dedup and the publish are one atomic fact, and redelivery after a worker crash is safe: the dedup id is either present, in which case the event is dropped, or absent, in which case it is applied.
Backpressure is the design’s response to a surge the system cannot immediately absorb. The naive response is unbounded queuing, which eventually exhausts memory and crashes. The senior response is bounded queuing with a deliberate shed policy: when the worker’s input queue is full, it stops fetching from Kafka, which causes the Kafka consumer lag to grow, which is visible and bounded by the topic’s retention. The downstream sees events later rather than never, and the system degrades rather than collapses. The shed policy is domain-specific: for market-data quotes, dropping the oldest unprocessed event for an instrument and processing only the latest is correct, because a stale quote is worse than a delayed fresh one; for ledger events, no shedding is acceptable and the queue must be sized to absorb the peak.
Idempotency at the downstream is the partner of dedup at the ingestion. Even with deduplication, a worker can crash after publishing to the downstream topic but before committing its offset, so on restart it reprocesses and republishes. The downstream consumer must therefore be idempotent on the event id, collapsing redeliveries just as the ingestion dedup layer does. The guarantee is at-least-once from ingestion plus idempotent collapse at the downstream, which gives the effect of exactly-once without the cost of distributed transactions. This is the engineering reality, and pretending to exactly-once across the boundary is a common mistake.
Backpressure policies per event category
The backpressure policy is per category because the three categories have different tolerance for shedding. Market-data quotes tolerate shedding: if the worker cannot keep up, it drops the oldest unprocessed quote for an instrument and processes only the latest. The policy is implemented as a per-partition bounded queue with a TTL on each event. When the queue is full, the worker compares the incoming event’s timestamp against the oldest event in the queue. If the incoming event is newer, the oldest is dropped and the incoming is enqueued. This ensures the worker always processes the most recent data, which is the correct behavior for a streaming system where staleness is worse than loss.
Reference-data updates do not tolerate shedding. Every update must be applied, and a backpressure situation must be resolved by adding capacity, not by dropping events. The worker for reference-data updates uses an unbounded queue, but the queue depth is monitored and alerts when it exceeds a threshold. The alert triggers a capacity review: the partition count may need to increase, or the worker’s per-partition processing capacity may need optimization.
Customer actions have a mixed policy. Out-of-order action duplicates are tolerated, but lost actions are not. The worker uses a bounded queue with a policy that rejects events older than a configurable timeout, because an action that has been waiting too long is likely in a retry loop that a newer action has already superseded. The rejection feeds back to the producer as an error that triggers the producer to retry the action with a fresh timestamp, so the action is eventually applied.
Monitoring the ingestion pipeline
The ingestion pipeline is monitored through three lenses: throughput, correctness, and health. Throughput monitoring tracks the per-partition event rate, the per-partition processing latency, and the per-partition consumer lag. The lag is the primary scaling signal: a growing lag on one partition indicates a hot key, and a growing lag across all partitions indicates insufficient capacity.
Correctness monitoring tracks the dedup collision rate, which should be near zero; a rising rate signals a producer redelivering aggressively or a dedup-key design bug. It also tracks the outbox publish success rate, which should be one hundred percent, and the downstream consumer’s ack rate, which indicates whether the downstream is keeping up.
Health monitoring tracks the worker’s CPU, memory, and JVM metrics, the dedup database’s connection count and query latency, and the Kafka cluster’s broker health. It also tracks the rebalance frequency and duration, which are the primary signals of instability in the consumer group.
The three lenses are displayed on a single dashboard that the on-call engineer checks first. The dashboard has a row per partition, with the throughput, lag, and error rate for each. A partition that shows lag growing and a non-zero error rate is the partition to investigate first.
Data model
The dedup table is the central entity, and its partitioning is the scaling decision. The outbox row ties dedup to publication.
The dedup row’s id is the producer-supplied UUID, and the table is hash-partitioned on it so the dedup load spreads. The outbox row carries the canonical payload and the downstream topic, and its commit is atomic with the dedup insert. The downstream event is the canonical form the rest of the platform consumes, and it carries the same id so downstream consumers can deduplicate idempotently.
Tradeoffs and alternatives
The monolithic ingestion service is the alternative the prompt is built around, and its failure modes are structural. Its single processing pool is the throughput ceiling, because adding capacity means a bigger machine. Its single queue is the single point of failure, because a backup blocks everything. Its single ordering scope forces a choice between parallelism and ordering, because one queue cannot be consumed in parallel without giving up per-key order. The partitioned design escapes all three: throughput scales with partitions, failure is per-partition, and ordering is per-key within a partition. The trade is operational complexity, managing partitions, rebalancing, and per-partition state, which the monolith avoids. For a system that must scale, the complexity is the cost of scaling, and the monolith’s simplicity is the cost of not scaling.
A single shared dedup table versus a partitioned one is the dedup tradeoff. The shared table is simpler and works at modest volume; it becomes the bottleneck as event rate rises, because every event contends for one table. The partitioned table scales the dedup load across the database’s parallelism, at the cost of partition management and the benchmark work to choose the partitioning strategy. Trade Republic’s benchmark showed hash partitioning on the id winning, which is the production choice.
Exactly-once versus at-least-once with idempotent collapse is the delivery guarantee tradeoff. Exactly-once across distributed consumers requires two-phase commit or transactional consuming and producing, which is expensive and operationally fragile. At-least-once with idempotent collapse at the downstream gives the same effect for the user at far lower cost, and it is the engineering reality. The trade is that every downstream consumer must implement idempotency, which is a discipline rather than a property.
A push-based intake where producers call an API versus a pull-based intake where workers read from external sources is an architectural tradeoff. The push-based intake, used for real-time feeds and partner integrations, puts the intake behind a load balancer and writes to Kafka. The pull-based intake, used for batch files and polling-based APIs, has workers that poll on a schedule. Both write to the same partitioned topic, so the downstream is uniform. The choice depends on the producer, not on the ingestion design.
Failure modes and what breaks at scale
A worker crash mid-processing is the primary failure mode and the one the design is built around. The worker has consumed an event from its partition and is processing it when it dies. Because the offset was not committed before the crash, the event is re-delivered to the new owner of the partition on rebalance. The dedup layer collapses the redelivery if the transaction had committed, or applies it if it had not, so no event is lost and none is doubled. The correctness rests on the outbox, which ties the dedup insert and the publish to one transaction.
A hot partition, where one key generates a disproportionate share of events, makes the owning worker the bottleneck while others idle. The detection is per-partition lag monitoring: a hot partition shows growing lag while others are stable. The mitigation is deliberate key splitting, where the hot key is spread across sub-partitions with a deterministic suffix, accepting that per-key ordering across the sub-partitions is lost. This is a scaling limit that must be monitored and acted on, not a property the design gives for free.
A rebalance storm, where workers repeatedly join and leave, causes continuous partition reassignment and processing pauses. The causes are often operational: unhealthy workers being restarted, or resource limits causing evictions. The mitigation is cooperative rebalancing, stable worker sizing, and a healthy worker lifecycle that does not flap. This is the operational cost of the partitioned model, and managing it is part of running the system.
Dedup table bloat, where the dedup store grows without bound because old ids are never purged, is a slow failure. The mitigation is a retention policy: ids older than the producer’s maximum redelivery window can be purged, because a redelivery after that window is not expected. For the partitioned dedup table, this is a partition-drop operation on old partitions, which is cheap. Without retention, the table grows until it degrades.
Downstream saturation, where a downstream topic’s consumers cannot keep up, causes the ingestion workers’ publishes to back up, which causes the workers’ input queues to fill, which triggers backpressure up to the intake. The system degrades by increasing latency rather than by losing events, as long as the Kafka topic retention is sufficient to hold the backlog. The failure mode to avoid is unbounded in-memory queuing, which crashes the worker; the bounded-queue-plus-shed policy is the defense.
Schema evolution for the ingestion pipeline
Events evolve as the business changes. A market-data quote may gain a new field for the exchange qualifier; a reference-data update may add a new attribute for the instrument’s ESG rating. The pipeline must handle schema evolution without blocking old producers or requiring a coordinated upgrade across all producers and consumers.
The canonical event format uses Protobuf with backward-compatible evolution rules. A new field is added with a new field number, and the old field number is never reused. The consumer reads only the fields it knows about and ignores unknown fields. The schema is registered in a schema registry, and the producer includes the schema id in the event envelope. The consumer fetches the schema from the registry on first encounter and caches it for subsequent events.
A production guard that a senior insists on is the transition period. When a new schema version is registered, the old version is not immediately retired. Both coexist for a transition window, typically several days, during which old producers continue to produce events with the old schema and new producers use the new schema. The consumer handles both during the transition. After the transition window closes, the old schema is retired and old producers must upgrade. The guard prevents an incident where a producer upgrade is rolled back but the new schema is already in use, causing the ingestion pipeline to reject events from the rolled-back producer. The transition window is set longer than the maximum expected rollback window.
Schema compatibility enforcement
The schema registry enforces a compatibility policy before accepting a new schema version. The policy is backward-transitive: a new schema must be able to read data written with any previous schema within the compatibility window. Backward compatibility means that a consumer written against the new schema can read events produced with the old schema. This is the standard policy for streaming systems where consumers are upgraded independently of producers.
When a new schema violates the compatibility policy, the registry rejects the registration and returns an error to the producer. The producer must fix the schema and resubmit. The rejection prevents a misconfigured producer from registering a schema that would break downstream consumers silently.
The schema registry is also part of the deployment pipeline. A new schema version is validated against the compatibility policy in the CI build, before it reaches the production registry. The validation catches incompatible changes early, before they can affect production events. A senior ensures that the schema validation is a required step in the CI pipeline, not an optional check that the team can skip during a hotfix.
Operational concerns
Observability keys off the partition. Track per-partition consumer lag, because lag is the primary scaling signal: growing lag on one partition means a hot key or a slow worker, and growing lag across all partitions means insufficient workers. Track the dedup collision rate, which should be low; a high rate signals a producer redelivering aggressively, which is worth investigating. Track the rebalance frequency and duration, because a flapping group is an operational problem.
Capacity planning keys off the partition count and the per-partition throughput. The partition count is chosen in advance for the expected peak, because changing it later requires rekeying and is disruptive. The worker count scales up to the partition count and is sized for the current load. The dedup table’s partition count is chosen for the dedup load. These are three independent scaling decisions, and conflating them, sizing the dedup table off the event rate without considering the partition parallelism, is a common error.
The producer contract is an operational concern that ingestion owns. Producers must supply a stable id for deduplication, must handle redelivery gracefully, and must respect backpressure when the intake signals it. Documenting and enforcing this contract is part of the system, because a producer that omits the id or that redelivers with a new id defeats the dedup layer. The ingestion service’s API should reject events without an id rather than silently inventing one.
Deployment safety means rolling worker deploys must not stall ingestion. The consumer group protocol handles this: workers are drained one at a time, partitions are reassigned, and the new workers pick up from the committed offsets. A rolling deploy thus looks like a brief per-partition pause, not a gap. The dedup table and the topics are independent of the workers, so they are not affected.
Defending the design to a Staff engineer
A Staff engineer will probe the monolith versus partitioned choice, and you should preempt it. The monolith fails structurally: one processing pool is the throughput ceiling, one queue is the single point of failure, and one ordering scope forces a choice between parallelism and order. The partitioned design escapes all three by making the partition the unit of scale, failure, and order. The trade is operational complexity, managing partitions and rebalancing, which is the cost of scaling and is worth paying for any system that must grow.
The correctness defense is at-least-once with idempotent collapse, tied together by the outbox. The worker writes the dedup id and the outbox event in one transaction, so the dedup and the publish commit together, and Debezium streams the event downstream. A worker crash redelivers the event, and the dedup layer collapses it if the transaction had committed or applies it if not. The downstream consumer is idempotent on the id, collapsing any redelivery from a crash before offset commit. The effect is exactly-once for the user, achieved without distributed transactions.
The scaling defense is partitioning as the primitive. Throughput scales with the partition count times the per-partition throughput, so adding workers up to the partition count adds capacity, and the partition count is chosen for the peak. The dedup table is partitioned on the id, so its load scales with the database’s parallelism. Hot keys are detected by per-partition lag monitoring and split deliberately. Rebalancing is managed by cooperative rebalancing and stable worker sizing. Each failure mode has a named mitigation.
The monolith fails because it has one of everything; the partitioned design scales because the partition is the unit of scale, failure, and order, and correctness rests on the outbox tying dedup to publication so that at-least-once delivery plus idempotent collapse gives the effect of exactly-once.
Common mistakes candidates make
- Designing a monolithic ingestion service. The documented failure mode. One processing pool is the ceiling, one queue is the single point of failure, and one ordering scope forces a choice between parallelism and order. Partition.
- Choosing a partition key that does not balance. A key that concentrates events on one partition creates a hot spot. Choose a key that distributes evenly and groups only what must be ordered.
- Deduplicating with an unpartitioned table. At scale, the single table is the bottleneck. Partition the dedup table on the id, and use hash partitioning, which the benchmark showed winning.
- Separating the dedup insert from the downstream publish. A crash between them can lose or duplicate the event. Tie them with the outbox so they commit in one transaction.
- Assuming exactly-once across distributed consumers. Exactly-once requires two-phase commit and is fragile. Use at-least-once with idempotent collapse at the downstream and make every consumer idempotent.
- Unbounded in-memory queuing for backpressure. That exhausts memory and crashes. Use bounded queues with a deliberate shed policy, and let Kafka lag absorb the backlog.
- Ignoring hot-key detection. A hot key unbalances the partitioning and makes one worker the bottleneck. Monitor per-partition lag and split hot keys deliberately.
- Omitting the producer id contract. A producer that omits the id or redelivers with a new id defeats the dedup layer. The intake should require and validate the id.
- Not setting a retention policy on the dedup table. The table grows unboundedly without pruning, degrading performance and consuming disk. Drop time-range partitions outside the producer’s maximum redelivery window.
- Using list partitioning on a computed key instead of hash partitioning on the UUID. List partitioning is slower, cannot use a primary key on the parent table, and has worse performance at scale. Use Postgres hash partitioning as the benchmark shows.
- Applying the same backpressure policy to all event categories. Market-data quotes tolerate shedding; reference-data updates do not. Use a per-category backpressure policy: shed for quotes, backpressure for reference data, bounded queue with timeout for customer actions.
- Not distinguishing the three event categories in the partition key design. All three use the same pipeline but need different partition keys and dedup policies. Quote by instrument for distribution, reference data by entity for ordering, customer actions by customer id for per-customer order.
What the interviewer asks next
Expect to be pushed on the partition count. The question: how do you choose it, and what happens if you guess wrong? The answer is that the partition count is chosen for the expected peak throughput divided by the per-partition throughput, with headroom, because changing it later requires rekeying the topic and is disruptive. If you guess low, you hit a throughput ceiling that adding workers cannot fix; if you guess high, you pay the overhead of empty partitions, which is modest. Over-provisioning is the safer error for a growing system.
A second follow-up: what happens when a worker crashes after publishing downstream but before committing its offset? The answer is that the event is redelivered to the new partition owner, the downstream consumer receives a duplicate, and the downstream’s idempotency on the event id collapses it. This is why every downstream consumer must be idempotent: at-least-once from ingestion is the guarantee, and idempotent collapse gives the effect of exactly-once.
A third, deeper probe: the dedup table grows forever. How do you bound it? The answer is retention: ids older than the producer’s maximum redelivery window can be purged, because a redelivery after that window is not expected. For the partitioned dedup table, this is a partition-drop operation on old partitions, which is cheap and online. Without retention, the table grows until its index and scan performance degrade. The retention window is a contract with the producer: it must not redeliver older than the window, or the dedup layer will fail to collapse the redelivery.
A fourth probe: how do you handle a poison-pill event that blocks the outbox publish for its partition? The answer is the same pattern documented in the Trade Republic outbox engineering blog: group the outbox messages by a key and retry failing groups independently. A poison pill in one message blocks only its group, not the entire outbox publish batch. The worker detects the poison pill by catching the deserialization or validation error, logs the event for investigation, and moves on to the next group. The poison-pill event is routed to a dead-letter queue for manual review.
A fifth probe: the producer contract requires an event id for dedup, but what happens when a producer sends events without an id? The intake rejects the event with a clear error message explaining the requirement. It does not silently generate an id, because a generated id breaks the client’s ability to retry with the same id. The rejection is a 400 error that the producer must handle by adding the id. The intake logs the rejection and monitors the rejection rate per producer, because a sudden increase signals a producer that has not updated its client to include the id.
Mastery Questions
Question
Why does a monolithic ingestion service fail at scale, and how does the partitioned design escape each failure mode?
Answer
The monolith fails structurally: its single processing pool is the throughput ceiling, its single queue is the single point of failure, and its single ordering scope forces a choice between parallelism and order. The partitioned design makes the partition the unit of scale, failure, and order. Throughput scales with the partition count, failure is per-partition, and ordering is per-key within a partition. The trade is operational complexity, which is the cost of scaling and is worth paying for any system that must grow.
Sources & evidence5 claims · 4 cited
Design based on Trade Republic engineering blog sources (src_tr_batchjobs, src_tr_partitioning, src_tr_outbox1) and interview context (src_tr_interview); design reasoning beyond sources is internal-reasoning and carries no claim.
- Trade Republic's horizontally scalable ingestion design uses partitioned Kafka topics feeding partitioned workers on Kubernetes, where each worker owns a disjoint partition slice and scales with the partition count, countering the documented monolithic-ingestion failure mode.verified
- The dedup table for event ingestion is hash-partitioned on the event UUID, because Trade Republic's benchmark found hash partitioning outperforms list partitioning on inserts with higher TPS and lower latency, and hash partitioning preserves the ability to put a primary key on the parent table.verified
- The monolithic ingestion service is the documented candidate failure: its single processing pool is the throughput ceiling, its single queue is the single point of failure, and its single ordering scope forces a choice between parallelism and ordering. The partitioned design escapes all three.verified
- The dedup insert and the downstream publish are tied in one PostgreSQL transaction via the outbox, so a crash between them cannot lose or duplicate the event; Debezium streams the outbox row downstream with the database's atomicity guarantee.verified
- Hot partition detection uses per-partition consumer lag monitoring: a growing lag on one partition indicates a hot key, and the mitigation is deliberate key splitting across sub-partitions with a deterministic suffix, accepting that per-key ordering across sub-partitions is lost.verified
Cited sources
- Scaling batch jobs for reliable and efficient processing · Trade Republic Engineering
- Postgres partitioning performance: Hash vs. List · Trade Republic Engineering (Sadeq Dousti)
- How to Pass the Trade Republic Software Engineer Interview (JobMentis) · JobMentis (interview outcome database)
- PostgreSQL + Outbox Pattern Revamped Part 1: Incidents and a new design pattern · Trade Republic Engineering (Sadeq Dousti)