On this path · Storage 3. Distributed Tracing Engine
Distributed Tracing Engine
A trace store with head and tail sampling, a Parquet columnar schema, Bloom-filter trace lookup, and TraceQL planning.
Learning outcomes
A log tells you that an error happened, and a metric tells you how often. Neither tells you which of the forty service calls inside one user request actually caused the latency spike. That question, the one that spans process boundaries, is the question a distributed tracing engine exists to answer.
After studying this page, you can:
- Explain why a trace store is split into collectors, ingesters, queriers, and a compactor, and why the read and write paths are decoupled.
- Specify the OTLP write contract and a TraceQL query contract in Protocol Buffers.
- Map the in-memory trace assembly map and Bloom filters, and the on-disk Parquet columnar schema with its row groups, column chunks, and footer.
- Describe how compaction merges small Parquet blocks, sorts row groups, and enforces retention.
- Diagnose the head versus tail sampling tradeoff, high-cardinality attribute explosion, partial traces from a collector crash, and multi-tenant isolation.
- Reason about garbage collection from span deserialization, columnar encoding CPU cost, and memory-mapped Parquet reads.
- Trace the 10x, 100x, and 1000x evolution, ending in cell-based architectures and a decoupled ingest log.
- Ground the design in Grafana Tempo’s Parquet block format, TraceQL, and Bloom-filter trace lookup.
Before we dive in
Picture a checkout request that crosses fifteen services. One of them is slow, and the customer abandons the cart. Each service emits its own logs and metrics, so each team opens its own dashboard, sees its own green checkmarks, and concludes the bug is somewhere else. The dashboards cannot disagree more and none can pinpoint the culprit, because none of them carries the one thing that would settle the argument: a single identifier threaded through every hop of that one request.
That identifier is the trace identifier, and the practice of threading it through every call is what forced distributed tracing into existence. A span records one operation: when it started, when it ended, which service performed it, and which parent span it belongs to. A trace is the tree of spans that share an identifier. Storing those trees is a different problem from storing logs or metrics. A span arrives spread across time, because a request fans out and converges; a trace is not complete until its last span lands, which may be seconds later. A query rarely asks for a time range of all traces, the way a metric query does; it asks for one specific trace by its identifier, or for the handful of traces matching some attribute. And the volume is brutal: a busy service emits millions of spans per minute, each carrying a bag of attributes.
The engine that solves this is shaped by those pressures. It must assemble spans into traces despite their staggered arrival, store them in a way that makes single-trace lookup cheap and attribute search possible, and do both at a write throughput that a row store cannot sustain. The shape it converges on is a columnar block format for storage, a Bloom filter for trace lookup, and a write path decoupled from the read path by a durable log. This page walks that shape from the topology down to the bytes.
Microservices topology and component interaction
The trace engine splits along the same line as every telemetry store: the place where spans arrive and are assembled, and the place where finished traces are read. Between them sits storage, and across the whole system runs the rule that recent data is mutable and held in memory while historical data is immutable and held in blocks.
A span’s journey starts at an instrumented application and reaches a collector receiver, whose only job is to accept spans over a wire protocol and forward them. The receiver hands spans to a distributor, which validates them against per-tenant limits, shards each trace by its trace identifier so that all spans of one trace land on the same partition, and writes them to a durable ingest log. Acknowledgement to the client happens only after the log confirms the write, which is what makes the write durable without replicating in-memory state.
From the log, two consumers read independently and therefore never block each other. The ingester (also called the live-store) consumes spans, assembles them into traces held in memory, and serves queries against that recent window, typically the last tens of minutes. The block-builder consumes the same log, organizes spans into Parquet blocks covering a configurable time window, and flushes them to object storage for long-term retention. Because both consume unique partitions sharded by trace identifier, neither needs to deduplicate against the other.
The querier sits on the read side. It receives requests from the query frontend, which shards the search space so multiple queriers work in parallel, and it answers a request by consulting both the ingester for recent data and object storage for historical blocks. The compactor runs on a schedule to merge small Parquet blocks into larger ones, deduplicate replicated spans, and delete blocks past retention.
Protocols and the write path
The dominant wire protocol is the OpenTelemetry Protocol, OTLP, which ships over both gRPC and HTTP. The receiver accepts the OTLP payload, and the distributor’s write path then sharding, rate limiting, and logging happen before any span touches memory. The write path’s unit of work is the span: it arrives carrying its trace identifier, its own span identifier, an optional parent span identifier, start and end timestamps, and a bag of attributes. The ingester groups spans by trace identifier into an in-memory trace and holds that trace open until it goes idle or a maximum live window elapses, at which point it flushes the assembled trace as part of a Parquet block.
The key property of the write path is that a trace is assembled over time, not written atomically. Spans of one trace may arrive seconds apart as the request fans out and converges. The ingester is the component that absorbs that stagger, holding a partial trace open and appending each span as it lands.
The read path and mutability
The read path mirrors the write path’s split. A query for a specific trace identifier asks the querier to check the ingester first, in case the trace is still live, then fall back to object storage. A TraceQL search across attributes fans out across both: the ingester for recent matches and the historical blocks for older ones, with results merged by the query frontend.
The mutability boundary is the spine of this topology. Live traces in the ingester’s memory are mutable: spans keep arriving and the trace keeps growing. Once a trace flushes into a Parquet block, it is immutable, read-only forever until compaction rewrites it or retention deletes it. This is what makes concurrent reads safe: a query against a historical block reads files no writer can touch, so a slow search can never be corrupted by an incoming write, and a write burst can never block a running query.
Concrete API schemas
The write contract carries a batch of resource-scoped spans; the query contract carries a TraceQL expression and a time range. Both are Protocol Buffer messages.
The asymmetry between a batched Export RPC and a streaming Query RPC is deliberate: writes are cheapest when many spans coalesce into one call, while reads are most responsive when the first matching traces stream back before the scan of every block finishes.
Low-level data structures and storage schema
The schema splits into two layers. In memory, the engine assembles spans into traces and indexes them for fast lookup. On disk, it freezes those traces into a columnar Parquet block whose layout makes single-trace fetch and attribute search cheap.
In-memory assembly and Bloom filters
The ingester’s central data structure is a map keyed by trace identifier, whose value is the list of spans that have arrived for that trace so far. Each incoming span is appended to the list for its trace, and the trace stays open until it goes idle or hits the maximum live window. This map is the only place where a trace is mutable.
A Bloom filter is the engine’s answer to the most common read: is this trace identifier present? A Bloom filter is a probabilistic structure that can say a trace is definitely absent or possibly present, never definitely present. That one-sided guarantee is exactly what the querier needs to skip blocks that cannot contain the requested trace without reading their Parquet data. The tradeoff is a false-positive rate: some blocks are read that turn out to hold nothing, controlled by sizing the filter against the expected trace count.
The Parquet columnar block
When a live trace flushes, its spans become rows in a Parquet block. Parquet is a columnar format: rather than storing a span’s fields contiguously as a row, it stores all values of one field together as a column. A query that filters on one attribute reads only that column, not the whole trace.
The block directory is a small constellation of cooperating files: the columnar data, the Bloom filters that index it, the trace-id-to-row-group map, and the metadata that gates visibility. The entity view below makes those ownership relationships explicit.
A small number of fields are stored as top-level intrinsic columns: the trace identifier, span identifier, parent identifier, service name, operation name, duration, and start time. These are the fields queried most often, so giving them their own columns makes those queries cheap. All other attributes live in a generic nested attributes column, and frequently queried attributes can be promoted to their own dedicated columns so TraceQL can read them without scanning the nested map.
A Parquet file ends with a footer that holds the schema and the metadata for every column chunk, including its offset, encoding, and compression. A reader opens the file, reads the footer first, and uses the offsets to seek directly to the column chunks it needs, which is why a columnar query does not pay the cost of reading the columns it ignores.
Compaction and retention
Blocks accumulate. The block-builder cuts a new block every few minutes, so a busy hour produces many small blocks, and querying across them means scanning many footers and Bloom filters. Compaction’s job is to merge small blocks into fewer larger ones, sort the row groups inside them so a single trace’s spans sit together, and delete blocks past retention.
During a merge the compactor reads the small blocks, sorts their spans by trace identifier and start time so that all spans of one trace sit contiguously, and drops duplicate spans that arrived through write path replication. Because a block is immutable and self-describing, retention is simply deleting a block directory: there is no live data to rewrite and no global index to rebuild.
Edge cases and chaos engineering
Head versus tail sampling
Sampling decides which traces a system keeps, because keeping every trace at high volume is rarely affordable. The choice is where in the pipeline the decision is made. Head sampling decides at the source, the moment a trace starts, typically by hashing the trace identifier against a rate. It is cheap and stateless, but it is blind: a head sampler that keeps one percent of traces keeps one percent of the slow traces and one percent of the error traces alike, so a rare failure that occurs in one trace out of a million is almost certainly dropped.
Tail sampling decides after the trace is complete, which is why it needs whole-trace assembly. A tail sampler waits until every span of a trace has arrived, inspects the full tree, and applies policies like keep traces with a server error, or keep traces slower than two seconds. Because it sees the whole trace, it can keep every interesting trace and drop only the boring ones. The cost is that the engine must hold each trace open until it is complete, which is exactly the in-memory assembly work the ingester does, and which is why tail sampling is run at the collector or ingester rather than at the source.
The tradeoff is the tradeoff of every sampling decision: cheap-and-blind versus expensive-and-aware. Production systems often combine them, head sampling at the edge to shed obvious volume and tail sampling at the collector to rescue the interesting traces that survive.
High-cardinality attribute explosion
Attributes are the bag of key-value pairs every span carries, and the same danger that haunts a metric label set haunts them: an attribute with unbounded values explodes the storage footprint. A user identifier or a request identifier as an attribute means a distinct value for nearly every span, which defeats the compression that columnar storage relies on, because a column of mostly-unique values cannot be dictionary encoded efficiently.
The defense starts at admission. The distributor can cap the size of any single attribute and truncate values that exceed it, which protects queriers from out-of-memory crashes when a single trace carries a few hundred kilobytes of attribute data. Beyond truncation, the engine relies on promoting only the small set of attributes that are actually queried often into dedicated columns, leaving the rest in the generic nested attributes column where they compress but scan slowly.
Collector crash mid-trace
A trace is assembled over time, which means a collector or ingester that crashes mid-trace leaves a partial trace behind. Spans that arrived before the crash live in the durable log and are recovered when the consumer restarts and resumes from its last committed offset, so nothing the distributor acknowledged is lost. But the in-memory assembly map was wiped by the crash, so the trace is reassembled from the log only from the point the consumer replays.
The harder case is a trace that never completes because its source died: the service that emitted the first spans crashed before emitting the rest. The ingester cannot know the trace will never finish, so it holds the partial trace open until the idle timer fires, then flushes it as a complete, if short, trace. Downstream, a query that fetches that trace id gets a tree with missing branches, which is the honest representation of a request that was cut short.
Multi-tenant isolation
A trace store shared by many tenants must isolate them. The standard mechanism is a tenant identifier carried in a request header, threaded through every operation so that each tenant’s traces land in a separate namespace under the object storage bucket and each query is scoped to one tenant. Per-tenant ingestion limits cap the rate and trace size each tenant can push, so one noisy tenant cannot starve the others or blow past the ingester’s memory budget. Because traces are sharded by trace identifier and the tenant prefix partitions the key space, one tenant’s data never interleaves with another’s in storage, and a compactor working one tenant’s blocks never touches another’s.
System hardening and programming realities
The ingester is an allocation hotspot. Every incoming span is deserialized from its OTLP wire form into an in-memory object, and deserialization allocates: each span becomes a struct, each attribute bag becomes a map, each timestamp becomes a fixed64. Under a sustained write burst this allocates objects faster than the garbage collector can reclaim them, and the resulting GC pressure shows up as latency spikes on the write path. The remedy is the usual one: pool span objects, avoid per-span map allocations by representing attribute bags as sorted slices until they are actually queried, and bound the live trace map so a burst cannot grow memory without limit.
The columnar encoding has its own CPU cost. Parquet compresses well, but compression is paid for at write time. Dictionary encoding builds a table of distinct values per column chunk and replaces each value with its index, which is cheap and effective for low-cardinality columns like service name. Run-length encoding collapses runs of identical values, which works well because spans of one trace share most fields. The CPU cost is the sort and encode pass the block-builder runs when it cuts a block, and the cost grows with the cardinality of the columns, which is another reason high-cardinality attributes hurt: they defeat dictionary encoding and force the writer into more expensive encodings.
On the read side, the querier serves historical blocks from memory-mapped Parquet files. The operating system maps a block file directly into the process address space, so reading a column chunk is a pointer dereference into the page cache, not a read syscall into a userspace buffer. Blocks are immutable, so a mapping never sees a torn read, and a query that scans many blocks streams these mappings without materializing the whole file, which is how a search over hours of data stays within memory.
Architectural evolution: 10x, 100x, 1000x
At 10x: single collector fan-in
The first saturation is the single collector. As span volume grows, one collector receiver becomes the bottleneck between the instrumented applications and the ingesters. The 10x response is to fan out the collectors behind a load balancer, each one stateless and interchangeable, sharding spans by trace identifier so all spans of one trace still land on the same downstream partition. The ingester’s in-memory map and Bloom filter absorb the read amplification, and compaction runs frequently enough to keep the live window small.
At 100x: tail sampling at scale, block splitting, Bloom sharding
At 100x, two things break at once. Tail sampling, which needs whole-trace assembly, cannot fit every live trace in one ingester’s memory, so the live window is split across a ring of ingesters, each owning a slice of the trace identifier space. Parquet blocks cut every few minutes grow numerous enough that querying across them scans too many footers, so block splitting balances block size against row group size, and compaction merges aggressively to keep the block count bounded.
The Bloom filters that make trace lookup cheap also grow, and a single filter per block becomes a bottleneck when thousands of blocks must be checked per query. The response is Bloom filter sharding: split each block’s filter across multiple sidecar files keyed by a hash of the trace identifier, so a querier checks only the shard that could hold its trace, cutting the filter read volume by the shard count.
At 1000x: cell-based, decoupled ingest-storage via a log
At 1000x, the coupling between the ingester and its in-memory traces is the limit. An ingester that holds live traces must be replicated for durability, which wastes memory when the same data lives in several heads. The macro-shift is decoupling ingest from storage: writes land in a durable, partitioned log first, and the ingester becomes a stateless consumer that assembles traces from the log. Durability now lives in the log, not in replicated heads, so the ingester scales freely and rebuilds any live window by replaying the log.
Beyond that, the architecture moves to cells. Each cell owns a slice of tenants with its own log, object storage bucket, and ring, coordinated by an eventual async fabric rather than a global synchronous one. A failure or a hot tenant is contained within its cell, and capacity is added by standing up a new cell rather than by growing a global one.
How Grafana Tempo does it
This design is the shape of Grafana Tempo. Tempo ingests spans, sorts their resources and attributes into columns in an Apache Parquet schema, and stores blocks in object storage for long-term retention. A trace’s journey matches the topology exactly: the distributor accepts spans over OTLP and other protocols, validates them against per-tenant limits, shards each trace by its trace identifier, and writes records to a Kafka-compatible durable log. Acknowledgement to the client happens only after the log confirms the write. From the log, the live-store consumes spans to serve recent queries while the block-builder consumes the same log to build Parquet blocks for object storage, the two reading independently without deduplication because traces are sharded by identifier.
Tempo’s on-disk format is a versioned Parquet block, currently vParquet4 by default with vParquet5 opt-in. A block is a directory in object storage containing a meta.json that records the block id, tenant, and time range and that makes the block visible to queriers only once it exists; a data.parquet holding the columnar trace data; Bloom filter sidecar files for efficient trace identifier lookup; and an index mapping trace identifiers to row groups. Intrinsic fields like service name and status code are top-level Parquet columns, while other attributes live in a generic Attrs column and can be promoted to dedicated columns for the attributes a deployment queries most.
TraceQL is Tempo’s query language, designed for selecting traces and using PromQL and LogQL-like syntax where possible. A query like one filtering on an HTTP status code reads only that status code column, not the entire trace, which is the columnar advantage Parquet provides. TraceQL requires the Parquet format, which is why the block format and the query language evolved together. The querier fetches Bloom filters and indexes from object storage to locate the traces within blocks efficiently, and the query frontend shards the search space across queriers that work in parallel and then concatenates their results.
The live-store holds recent trace data in memory and serves queries during the window between ingestion and block availability in object storage, periodically flushing traces to a local write-ahead log in Parquet format for search and metrics queries. The compactor, increasingly succeeded by a backend scheduler and worker architecture, merges blocks, deduplicates spans, and expires data past retention by deleting block directories.
Mastery Questions
Question
Why does a trace store decouple its write path from its read path with a durable log, rather than letting the ingester write straight to object storage?
Answer
Because acknowledgement to the client must be durable the instant spans are accepted, but assembling traces and building Parquet blocks take time. Writing to a durable log first lets the distributor acknowledge once the log confirms the write, then lets the ingester and block-builder proceed at their own pace. A slow compactor cannot stall an incoming write, and a slow query against blocks cannot stall a write, because the two paths meet only inside the log.
Sources & evidence8 claims · 7 cited
All eight mechanism and operational claims are backed by fetched primary sources (Grafana Tempo architecture, components, config, block format, TraceQL, distributor docs; OTLP proto; Apache Parquet format). The 10x/100x/1000x evolution is internal-reasoning extrapolation, not asserted as sourced fact.
- The OTLP ExportTraceServiceRequest holds a repeated list of ResourceSpans, where each ResourceSpans wraps a Resource and repeated ScopeSpans, each ScopeSpans wrapping repeated Spans carrying trace id, span id, parent span id, start and end time in unix nanoseconds, and attributes.verified
- Tempo distributor can cap the size of any single attribute and truncate values that exceed it, because large span attributes have caused querier out-of-memory errors when fetching even small traces.verified
- Tempo blocks carry Bloom filter sidecar files that give the querier a fast probabilistic trace id membership test, and an index that maps trace ids to row groups within data.parquet.verified
- Tempo promotes frequently queried attributes from the generic Attrs column into dedicated Parquet columns configured centrally in storage.trace.block, applied by all block-producing components, so TraceQL can read them without scanning the nested attribute map.verified
- Tempo shards traces by trace id into a Kafka-compatible durable log, and the live-store and block-builder consume from it independently without deduplication, with the distributor acknowledging the client only after the log confirms the write.verified
- A Tempo block is invisible to queriers until its meta.json file exists, a property the block-builder relies on to make flushes atomic.verified
- Grafana Tempo stores trace data in Apache Parquet columnar format, where each row represents a span with intrinsic columns for trace id, span id, service name, and duration, and a generic nested column for other attributes.verified
- TraceQL is Tempo query language for selecting traces, uses PromQL and LogQL-like syntax where possible, and requires the Parquet columnar block format, which is the default for Tempo.verified
Cited sources
- OpenTelemetry Protocol trace_service.proto · OpenTelemetry Authors
- OpenTelemetry Protocol trace.proto Span definition · OpenTelemetry Authors
- Configure Tempo · Grafana Labs
- Tempo distributor and ingestion limits · Grafana Labs
- Tempo block format · Grafana Labs
- Tempo architecture · Grafana Labs
- TraceQL query language · Grafana Labs