Market Data Ingestion and Normalization

Learning outcomes

A brokerage does not generate prices. It receives them, from dozens of exchanges and data vendors, each speaking a slightly different dialect of the same idea, and it must turn that babel into one clean stream its products can use. The documented Trade Republic candidate failure is to assume all exchanges provide data in a consistent format. This page teaches why that assumption is fatal and how a real ingestion pipeline is built to absorb the inconsistency.

After studying this page, you can:

• Explain why market data arrives in incompatible shapes and why a normalization layer is mandatory, not optional.
• Describe FIX and FAST, what FIX tags carry, and how a normalized internal quote model hides vendor differences from downstream services.
• Handle out-of-order, late, and duplicate ticks correctly using sequence numbers and per-symbol ordering.
• Defend a horizontally scalable, partitioned ingestion design against the monolithic-ingestion failure mode.
• Connect raw quotes to derived metrics like yield to maturity, and explain why customers care about derived values more than raw prices.

Before we dive in

A customer opens a bond detail page and sees a yield number. That number is not sent by any exchange. The exchange sends a price quote, in a format the exchange chose, and the brokerage computes the yield from it, in real time, before the page renders. The gap between what the exchange sends and what the customer sees is the entire market-data ingestion problem, and it is wider than most engineers assume before they touch it.

The problem exists because exchanges are independent organizations that grew up independently. Each chose its own wire format, its own field semantics, its own trading calendar, its own tick size, and its own way of identifying the same security. The Xetra order book, a US options feed, and a bond price vendor all describe trades and quotes, but they describe them in dialects that share vocabulary and disagree on almost every detail. A brokerage that connects to several of them cannot treat them as one source. It must build a layer that translates each dialect into one internal language, and that layer must be correct, because every price the customer ever sees flows through it.

Two engineering consequences follow, and both are tested in the interview. The first is that normalization is where correctness is won or lost. A field misread here, a sign flipped, a scale misapplied, and every downstream price is wrong in a way that is expensive to detect. The second is that ingestion is where scale is won or lost. Market data arrives in bursts, especially at the open and close, and a pipeline that holds together on a quiet day can collapse on a volatile one. The documented Trade Republic failure mode, building a monolithic ingestion service instead of a horizontally scalable one, is a failure to take the second consequence seriously.

Breaking it down

1. Why every exchange looks different

Start with the most basic question an interviewer uses to separate real understanding from memorization: why do exchanges not just agree on one format? The answer is history and incentive. Exchanges predate the brokers that consume their data, often by decades. Their formats encode decisions made when their systems were built, and changing a format breaks every downstream participant, which is expensive for the exchange and risky. There is no central authority that can force convergence, and the exchanges have no strong incentive to spend money making a competitor’s life easier. So the dialects persist.

The differences are concrete and they bite. Field naming differs: one feed calls the price LastPx, another calls it Price, another trade_price. Field semantics differ: one feed reports prices in the local currency’s minor units, another in major units, and a bonds feed may report price as a percentage of par. Symbol identifiers differ: the same security may be identified by an ISIN, a local exchange symbol, a ticker, or a vendor-specific code, and the mapping between them is itself a maintained dataset. Trading calendars differ, tick sizes differ, and the rules for what constitutes a valid quote differ.

The FIX protocol was invented to tame some of this, and it is the lingua franca of institutional trading, but it does not eliminate the differences. FIX standardizes the message envelope and a large set of numbered tags, but each exchange uses a subset of tags, extends FIX with vendor-specific tags, and applies its own semantics. So even FIX-speaking feeds disagree in practice. The candidate who assumes FIX means consistency has misunderstood what FIX standardizes and what it leaves to the vendor.

Assuming consistent exchange formats

The documented Trade Republic feedback names this as a real failure mode. The candidate who designs ingestion as if one parser handles all feeds will produce a pipeline that silently corrupts prices the moment a second feed is connected. The correct posture is to assume every feed is a separate dialect with its own adapter, and to treat the normalized internal model as the only thing downstream ever sees.

2. The normalized internal quote model

The defense against dialect chaos is a single normalized internal model and one adapter per feed. Every incoming message, whatever its source, is parsed by a feed-specific adapter into the same internal quote structure. Downstream services see only that structure and never the raw feed, so a new feed can be onboarded by writing one adapter, with no change to anything downstream.

The internal model must be rich enough to express what every feed can say and disciplined enough to prevent ambiguity. A minimal quote carries a symbol, resolved to an internal instrument id; a price in a fixed, normalized currency and scale; a quantity or size; a side, bid or ask or trade; a source venue; and an exchange-assigned sequence or timestamp for ordering. The discipline is in the scale and the identifier. Prices are normalized to a single decimal convention internally, so a downstream service never has to wonder whether a value is in minor or major units. Symbols are resolved to internal instrument ids at the adapter boundary, so downstream never sees a raw exchange symbol and never has to know the mapping.

The internal quote carries a normalized price, a resolved instrument id, and the exchange sequence used for ordering.

The choice to normalize at the adapter boundary, rather than letting each service interpret raw feeds, is the single design decision that keeps a market-data system maintainable. It localizes the per-feed knowledge to one place, the adapter, and makes the rest of the system feed-agnostic. When a feed changes its format, which happens, one adapter is updated and nothing else moves. When a new feed is added, one adapter is written and nothing else moves. The cost is the adapter layer itself, and the benefit is a downstream that never breaks on feed churn.

3. Out-of-order, late, and duplicate ticks

Networks are not FIFO. A quote sent first can arrive second, because it took a different path or was retried. A late tick can arrive minutes after the market moved on. A duplicate can arrive when a feed’s transport retries a delivery the sender already counted as done. A correct ingestion pipeline must handle all three, because a price computed from a misordered or duplicated stream is wrong, and wrong prices in a brokerage have direct financial consequences.

The tool for all three is the sequence number, assigned by the source. Each tick carries the source’s monotonically increasing sequence, and the pipeline uses it to decide what to do with an arriving tick. If the sequence is older than the latest applied sequence for that symbol, the tick is stale and is dropped. If the sequence equals one already applied, the tick is a duplicate and is dropped. If the sequence is the next expected one, it is applied. If a gap appears, a missing sequence that has not arrived, the pipeline must decide whether to wait for a retransmit or to advance and accept the gap, a tradeoff between latency and completeness that depends on the feed’s reliability guarantees.

The subtlety an interviewer probes is whether you drop a stale tick silently or log it. Silent drops hide data loss. The defensible design is to drop the tick from the live stream, because applying it would move the price backwards, but to count drops and gaps in metrics, so that a feed degrading into a stream of stale and duplicate ticks is visible in operations before a customer notices a frozen price.

4. Per-symbol ordering

Sequence numbers are per source, but the brokerage often needs ordering across sources for the same symbol. A stock listed on two exchanges produces two streams, each internally ordered by its own sequence, and the pipeline must merge them into one ordered view of that symbol’s price. This is per-symbol ordering, and it is the layer where cross-source correctness lives.

The hard part is that there is no single global sequence across sources. The pipeline cannot simply sort by sequence number, because two sources use independent sequence spaces. Instead, it orders within each source by that source’s sequence, and across sources by a merge policy, typically by the exchange timestamp adjusted for a known clock skew, or by arrival order with a grace window for stragglers. The merge policy is inherently approximate, because the sources do not share a clock, and a senior engineer’s job is to pick the approximation whose failure mode is least bad. Arrival-order merging with a short grace window fails gracefully by occasionally ordering a straggler late, which is preferable to timestamp merging, which can silently reorder when clocks drift.

This is the same family of problem as outbox ordering across pods: when the clocks disagree, you cannot sort by wall time, and you need a logical or per-source-sequence-based discipline. The connection is worth making explicitly in an interview, because it shows you recognize the pattern and are not reinventing it for market data.

5. Monolithic versus horizontally scalable ingestion

Ingestion has a characteristic load shape: low most of the day, with sharp bursts at the open, the close, and on news events. A pipeline sized for the average rate dies in the burst. The documented Trade Republic failure mode is building a monolithic ingestion service, one process that handles all feeds and all symbols, which cannot absorb the burst because it cannot scale out.

The fix is to partition. Ingestion is naturally partitionable by symbol and by feed, because ticks for different symbols are independent. A horizontally scalable design assigns symbol or feed partitions to independent workers, so that adding workers increases throughput. Trade Republic’s own batch-job scaling work describes this exact pattern: partitioned workers on Kubernetes, processing work reliably and efficiently at scale, rather than a single monolith that becomes a bottleneck and a single point of failure.

Partition by the independence of the work

The reason partitioning works for ingestion is that ticks for different symbols do not depend on each other. Independence is what makes horizontal scaling safe: each worker owns its partitions and never coordinates with the others. Where work is independent, scale out; where work must be ordered, keep it on one sequenced worker. The mistake in the monolith is forcing independent work through one serialized throat.

The partitioning decision interacts with the per-symbol ordering from the previous section. If a symbol’s ticks must be ordered, then all ticks for that symbol must land on the same worker, so the partition key is the symbol, not a random shard. This is the same key-ordering principle that governs Kafka partitions: all messages for a key go to one partition and are ordered within it. The ingestion design and the Kafka design agree because they solve the same ordering problem.

6. Derived metrics, from raw quotes to yield

The customer does not want a raw price. The customer wants a decision-relevant number. For a bond, that number is the yield to maturity, computed from the price quote, the coupon, the maturity, and the time to cash flow. Trade Republic’s published architecture for real-time bond yield is the cleanest illustration: a Kafka Streams topology consumes raw quotes, computes the yield to maturity per bond identifier, and sinks the result to both Redis, for live client fan-out, and a Kafka topic, for historical charts.

The architecture teaches two lessons that generalize beyond bonds. The first is that the normalized quote stream is a substrate for derived computation, not a final product. Each product computes what its customers care about: yield for bonds, implied volatility for options, index values for baskets. The ingestion pipeline’s job is to deliver a clean, ordered, normalized quote stream, and the derivation jobs’ job is to turn it into product-specific metrics. Keeping these layers separate means a new product can add a new derivation without touching ingestion.

The second lesson is the fan-out discipline. The yield pipeline uses Redis sharded pub/sub, not plain pub/sub, because non-sharded pub/sub broadcasts every message to every Redis node, which destroys scalability. A single Redis connection per service instance then fans updates to all connected WebSocket clients for that instance, and the service tears down the subscription when the last client for a bond leaves. This is the pattern that serves real-time data to millions of clients without the fan-out cost growing with the client count, and it depends on a clean ingestion stream as its input.

Common mistakes candidates make

• Assuming all exchanges provide data in a consistent format. This is the documented, named failure mode. Every feed is a dialect with its own adapter. The normalized internal model is the only thing downstream sees.
• Building a monolithic ingestion service. A single process handling all feeds and symbols cannot absorb the open and close bursts and is a single point of failure. Partition by symbol or feed and scale horizontally.
• Ordering by wall-clock timestamp across sources. Sources do not share a clock. Use per-source sequences for within-source order and a documented merge policy for across-source order.
• Dropping stale and duplicate ticks silently. Drops and gaps must be counted in metrics, or a degrading feed is invisible until a customer sees a frozen price.
• Confusing FIX with consistency. FIX standardizes the envelope and many tags, but each vendor uses a subset and extends it. FIX does not remove the need for per-feed adapters.
• Letting downstream services see raw feed formats. Normalize at the adapter boundary. The moment a downstream service interprets a raw field, feed churn becomes a distributed maintenance problem.

What the interviewer asks next

A common follow-up is to push on backpressure. The question: the open burst produces ticks faster than the Kafka Streams yield job can compute. What happens? The strong answer is that Kafka partitions buffer the burst, the stream job processes its partition at its own rate, and consumer lag is the observable signal; if lag grows unboundedly, the job is under-provisioned and must scale its consumer parallelism up to the partition count. A second follow-up: how do you guarantee the customer never sees a yield computed from a stale price? The answer ties back to ordering, the yield job consumes an ordered, deduplicated stream, so the yield it emits always reflects the latest applied quote for that bond.

Mastery Questions

Question

Why is assuming all exchanges provide data in a consistent format a fatal design error?

Question

How does an ingestion pipeline classify an arriving tick using its source sequence number?

Question

Why is per-symbol ordering across multiple sources approximate, and which merge policy fails most gracefully?

Question

Why does partitioning ingestion by symbol enable safe horizontal scaling, and why is the symbol the partition key?

Question

In the yield-to-maturity pipeline, why is Redis sharded pub or sub used instead of plain pub or sub, and when is a subscription torn down?

Answer

Exchanges grew up independently and each chose its own wire format, field semantics, symbol identifiers, tick sizes, and calendars. Even FIX-speaking feeds disagree because each vendor uses a subset of tags and extends with vendor-specific tags. Treating feeds as uniform produces a pipeline that silently corrupts prices the moment a second feed is connected. The fix is one adapter per feed and one normalized internal quote model that downstream ever sees.

Answer

Compare the tick's source sequence to the last applied sequence for that symbol. If it is older, the tick is stale and dropped; if it equals an applied sequence, it is a duplicate and dropped; if it is the next expected or a gap is accepted, it is applied; if there is a gap, the pipeline may wait for retransmit or advance after a timeout. Drops and gaps are counted in metrics so a degrading feed is visible.

Answer

There is no global sequence across sources, so cross-source order cannot be exact. Timestamp merging can silently reorder when clocks drift. Arrival-order merging with a short grace window for stragglers fails gracefully by occasionally ordering a straggler late, which is less harmful than silent reordering. The pattern mirrors outbox ordering: when clocks disagree, do not sort by wall time.

Answer

Ticks for different symbols are independent, so they can be processed in parallel without coordination. Partitioning by symbol lets independent workers own disjoint symbol sets and scale out. The symbol must be the partition key, not a random shard, because a symbol's ticks must be ordered, so all ticks for one symbol must land on one worker. This matches Kafka key-ordering: all messages for a key go to one ordered partition.

Answer

Plain pub/sub broadcasts every message to every Redis node, which destroys scalability at high tick rates. Sharded pub/sub routes a key's messages to one node. On the first WebSocket subscription for a bond a Redis subscription is created; when the last client for that bond leaves it is torn down, keeping the system lightweight. This keeps fan-out cost independent of client count.

Question

Why is assuming all exchanges provide data in a consistent format a fatal design error?

Answer

Exchanges grew up independently and each chose its own wire format, field semantics, symbol identifiers, tick sizes, and calendars. Even FIX-speaking feeds disagree because each vendor uses a subset of tags and extends with vendor-specific tags. Treating feeds as uniform produces a pipeline that silently corrupts prices the moment a second feed is connected. The fix is one adapter per feed and one normalized internal quote model that downstream ever sees.

Prev1 / 5Next