Time-Series Database Internals

Learning outcomes

A general-purpose database optimizes for flexible reads over changing rows. A time-series database faces the opposite problem: a flood of appends that almost never change, queried mostly over time ranges and label filters. That single difference reshapes every layer below.

After studying this page, you can:

• Explain why the read and write paths are separated by an immutable block, and name the services on each side.
• Specify the write and query contracts for a TSDB.
• Encode timestamps with delta-of-delta and values with XOR, and explain why both compress.
• Map the on-disk block layout, the symbol table, and the postings-based inverted index.
• Describe how a compactor merges blocks, deduplicates samples, and enforces retention.
• Diagnose cardinality explosions, crash recovery through WAL replay, and out-of-order writes.
• Trace the 10x, 100x, and 1000x evolution, ending in decoupled ingest-storage and cells.
• Ground the design in Prometheus’s TSDB format and Grafana Mimir and Thanos.

Before we dive in

Suppose you stored one million metrics, each emitting a sample every fifteen seconds, in a row-oriented database. Every insert writes a row carrying the full label set, the timestamp, and the value. Eighty percent of each row is redundant: the same labels repeat, the timestamps march in lockstep, and the values change by small deltas. A row store pays for that redundancy in disk, in memory, and in the cache misses that follow.

The insight that forced the time-series database into existence is that telemetry is regular. Timestamps advance by a near-constant interval, so the second difference of the timestamp is usually zero. Values drift slowly, so consecutive values of one series share most of their bits. Label sets repeat identically across thousands of samples. A storage engine that exploits this regularity, instead of treating each sample as an island, can shrink a sample from tens of bytes to roughly one.

But compression is only half the story. The other half is that a time-series workload is write-heavy and append-only, which means data, once written, never changes. That invites an architecture the row store cannot afford: write incoming samples into a small mutable head, then periodically freeze that head into an immutable block. Once frozen, a block is never edited; it is only read, merged into a larger block, or deleted when its retention expires. Immutability is what makes both compression and concurrent reads safe, and it is the property the entire topology is built to preserve.

Microservices topology and the immutable block

The TSDB splits cleanly along the line where mutable memory becomes immutable storage. On the write side, the ingester owns a mutable head; on the read side, the querier and store-gateway read immutable blocks. Nothing crosses back.

The ingester is the only service that mutates data. It accepts writes into an in-memory head, persists each to a write-ahead log, and every fixed interval (two hours in the reference implementation) cuts the head into a block and uploads it to object storage. The store-gateway is a stateless cache over those object-storage blocks: it serves historical queries by fetching and caching block chunks. The compactor runs as a singleton per block stream and merges smaller blocks into larger ones. The querier fans out to the ingester for recent data and the store-gateway for historical data, then merges.

The mutability constraint is absolute. The head is mutable; everything else is immutable. A query that touches recent data reads the head under a lock that allows concurrent append, but a query against a block reads files that no writer can touch. This is why a slow analytical query cannot block an incoming write, and why a write burst cannot corrupt a running query.

Network and storage boundaries

Writes arrive over gRPC as Snappy-compressed Protocol Buffers, the same contract the ingestion front uses. The ingester-to-object-storage boundary is HTTPS multipart PUT, async and retryable. The store-gateway reads blocks either by listing the bucket or by consulting a bucket index that records which blocks exist, so a querier does not pay the cost of listing a bucket on every query.

Write and read lifecycles

A write reaches the ingester already rate-limited and sharded by the ingestion front. The ingester appends it to the WAL and fsyncs, then adds the sample to the head chunk of the matching series. When the head’s two-hour window closes, the ingester compacts its in-memory chunks into a block directory (a chunks subdirectory, an index file, and a metadata file), uploads it, and truncates the WAL segments it no longer needs.

A read asks the querier for a range. The querier splits the range by the head boundary: the recent portion goes to ingesters, the historical portion to the store-gateway. Each returns partial series, which the querier merges by timestamp. Because blocks are immutable, the store-gateway can cache them aggressively and serve them from memory-mapped files without copying.

Concrete API schemas

The write contract carries the tenant, the series identity, and the samples; the query contract carries a range and a label matcher.

The split between a batched Write RPC and a streaming QueryRange RPC is deliberate: writes are cheapest when coalesced, while reads are most responsive when the first partial results stream back before the scan finishes.

Low-level data structures and on-disk schema

The XOR chunk: delta-of-delta and XOR encoding

The compression that defines a TSDB operates on the chunk, the unit that holds a contiguous run of one series’ samples. The reference encoding stores the first timestamp and value verbatim, then encodes every later timestamp with delta-of-delta and every later value with XOR.

For timestamps, the raw value is regular: it advances by a near-constant interval. The first difference removes that constant, and the second difference, the delta-of-delta, is usually zero.

The encoder then writes this residual with a variable-length code: the most common residuals cost a single bit, and progressively rarer, larger residuals cost progressively more bits. When samples arrive at a fixed interval, almost every timestamp costs one bit instead of eight bytes.

For values, the encoder stores the bitwise XOR of consecutive doubles. Slowly-drifting values share a long common prefix of significant bits, so the XOR tends to have leading and trailing zero runs. The encoder records the length of those zero runs and only the meaningful middle bits, again with a variable-length code. The first value is stored raw; each later value is stored as the XOR against the previous one.

The combined effect is that a sample, tens of bytes as a raw row, compresses to on the order of a byte. Both encodings are causal: they depend only on the previous sample, so a chunk can be decoded sequentially from its start, which is exactly how a range query scans it.

The block layout and the inverted index

When the head cuts, its chunks are written into a block directory alongside an index that lets a query find them by label.

The index file is an inverted index. It stores a sorted symbol table of every distinct label string (so each name and value appears once, referenced by an integer), a series section that maps each series to its labels and the chunks holding its samples, and postings lists that map each label name-value pair to the sorted list of series ids carrying it.

A query like “job equals api” becomes a postings-list lookup, an intersection or union across multiple matchers, and then a fetch of the chunks those series point to. The symbol table is what keeps the index small: a label value repeated across a million series is stored once, not a million times.

Compaction mechanics

Blocks accumulate. The compactor’s job is to merge many small blocks into fewer large ones, deduplicate samples that were replicated, reconcile out-of-order data, and delete blocks past retention.

Compaction takes a set of blocks that cover adjacent or overlapping time ranges and rewrites them into a single block. During the merge it sorts samples by series and timestamp, so out-of-order arrivals from different ingesters land in the correct position, and it drops duplicate samples that share a series and timestamp (the inevitable consequence of write-path replication). A two-level scheme is common: a first pass compacts blocks down to a medium size, a second pass compacts those into large blocks covering wide ranges, and a downsampling pass may aggregate old, high-resolution data into lower resolution so that a year-long query does not scan a year of fifteen-second samples.

Retention is enforced by deleting blocks whose max timestamp falls before the retention cutoff. Because blocks are immutable and self-describing, deletion is just removing a directory; there is no tombstone, no rewrite of live data, no index rebuild.

The reason deletion is just removing a directory is that a block is immutable and self-describing: nothing else references its internals, so there is no index to rebuild and no live data to rewrite.

Edge cases and chaos engineering

High-cardinality explosion

The same cardinality pressure that strains the ingestion front is even more dangerous here, because the index is where it accumulates. A label with unbounded values turns one metric into millions of series, and each series becomes an entry in the symbol table, a row in the series section, and an id in postings lists. The index file balloons, query planning slows as postings lists grow, and the store-gateway’s memory cache churns.

The defense begins at admission (a tenant series cap enforced upstream) but continues in storage. The index is sharded: rather than one global index, the series space is partitioned (commonly by tenant and by a hash of the label set) so that no single index file must hold every series. Postings lists are intersected lazily, starting from the matcher expected to select the fewest series, so a query that narrows quickly never materializes a huge intermediate list.

Crash recovery and WAL replay

An ingester crash mid-head leaves in-memory chunks unwritten and the WAL ahead of them. On restart the ingester replays its WAL segments in order, rebuilding each head chunk from the records. Because every accepted write was fsynced to the WAL before acknowledgement, replay restores the head exactly to the last acknowledged sample; nothing the distributor confirmed is lost. WAL segments older than the last flushed block are truncated, since the corresponding data already lives in an uploaded block. If a corrupt segment is encountered, replay stops at the corruption boundary, losing only the unacknowledged tail.

Out-of-order and late data

Samples sometimes arrive out of order, from a delayed agent or a retried batch. The head accepts them by inserting into the chunk at the correct position, but heavy out-of-order traffic fragments chunks and degrades compression. The compactor is the eventual reconciler: when it merges blocks it sorts everything by timestamp, so transient disorder in the head becomes clean order in the block. This is why the architecture tolerates mild out-of-order writes without special handling, but caps how far out of order a single write may be, to bound the head’s reordering cost.

System hardening and programming realities

Memory and garbage collection

The ingester’s head is the allocation hotspot. Each incoming sample appends to a head chunk, and if that chunk were a growing slice it would reallocate and copy under GC pressure. The remedy is a fixed-capacity head chunk per series: it is allocated once, filled by index, and sealed when full, so the steady-state allocation on the write path is near zero. The series map is sharded to keep lock contention low under concurrent append.

Memory-mapped files and zero-copy reads

On the read side, the store-gateway serves historical blocks from memory-mapped files. The operating system maps a block file directly into the process address space, so reading a chunk is a pointer dereference into the page cache, not a read syscall into a userspace buffer. Blocks are immutable, so a mapping never sees a torn read. Queries that scan large ranges stream these mappings without materializing the whole block, which is how a query over a year of data stays within memory.

Architectural evolution: 10x, 100x, 1000x

At 10x: the head and the mmap

The first saturation is the single ingester head. As series count grows, the head’s memory footprint and the lock on the series map contend. The 10x response is sharding the head map and memory-mapping the sealed chunks so the read path stops copying. Compaction runs more frequently to keep the head small, and the store-gateway’s block cache absorbs the read amplification.

At 100x: the bucket and the store-gateway

At 100x, the object-storage bucket itself becomes a bottleneck: listing it per query is too slow, and a single store-gateway cannot cache every block. The response is a bucket index that records which blocks exist and their time ranges, so queriers skip the listing entirely; and sharding the store-gateway across a hash ring so each gateway caches a slice of blocks. Compaction becomes parallel across disjoint block streams, each handled by its own compactor instance.

At 1000x: decoupled ingest-storage and cells

At 1000x, the coupling between an ingester and its head is the limit. An ingester that holds both the WAL and the head must be replicated for durability, which wastes memory when the same data lives in three heads. The macro-shift is decoupled ingest-storage: writes land in a durable, partitioned log (a Kafka-like ingest store) first, and the ingester becomes a stateless consumer that builds heads from the log. Durability now lives in the log, not in replicated heads, so the ingester can scale freely and rebuild any head by replaying the log. Beyond that, the architecture moves to cells, each owning a tenant slice with its own log, bucket, and ring, coordinated by an eventual async fabric rather than a global one.

Notice what the log removes: the ingester no longer needs a replicated head for durability, so three copies of recent data in memory collapse to one, and any failed ingester is rebuilt by replay rather than by copying a peer’s state.

How Grafana Mimir and Prometheus do it

This design is the shape of the Prometheus TSDB and its multi-tenant extension, Grafana Mimir. The on-disk format described here is the Prometheus TSDB: a block is a set of immutable files in a ULID-named directory holding a chunks subdirectory, an index file, a metadata file, and optional deletion and no-compact marks. The index file stores a sorted symbol table plus a series section, then label-index and postings sections terminated by a table of contents. The chunks are XOR-encoded with the delta-of-delta timestamp and XOR value scheme.

The WAL is a sequence of numbered segments capped by default at 128 megabytes, written into 32-kilobyte pages, borrowing the page and record encoding from the LevelDB and RocksDB write-ahead logs. The ingester stores the most recent samples in memory and on disk, cutting a two-hour block, uploading it to object storage, and replaying the WAL on restart.

Mimir extends this into a horizontally scalable, multi-tenant, long-term-storage system. Its ingester ring uses a configurable token count with a spread-minimizing strategy and a replication factor, so a distributor can shard and replicate each series across ingesters. Object-storage blocks are compacted by a compactor that must run as a singleton per block stream per bucket, with multiple compactors allowed only on disjoint streams. Thanos applies the same object-storage block model and adds a two-level compaction with downsampling, so a long-range query aggregates old data rather than scanning it at full resolution. Together these systems are the concrete instance of the principle that a TSDB is built to exploit regularity and preserve immutability, because those are the two properties that make a write-heavy, append-only workload compressible and safely concurrent.

Mastery Questions