On this path · Storage 2. Distributed Log Aggregation
Distributed Log Aggregation
A log engine with label-driven ingestion, chunk streaming, and decoupled read and write paths.
Learning outcomes
A log line is not a metric. It is a long, irregular string whose meaning lives in its words, not in a numeric value. That one difference is why a log aggregation engine refuses to index log content and instead indexes only the labels that name a stream, and why every layer of the system, from the write path to the chunk file, is shaped around that refusal.
After studying this page, you can:
- Name the services on the write and read paths and explain why read and write engines are decoupled.
- Specify the Push and Query contracts for a label-indexed log engine.
- Map the in-memory structures, the on-disk chunk file, and the inverted label index.
- Describe how a compactor merges index files, deduplicates, and enforces retention.
- Diagnose high-cardinality explosions, crash recovery through WAL replay, and multi-tenant isolation.
- Reason about GC pressure, batching, and zero-copy reads on both paths.
- Trace the 10x, 100x, and 1000x evolution toward cells and decoupled ingest-storage.
- Ground the design in Grafana Loki’s distributor, ingester, querier, query-frontend, index-gateway, and compactor.
Before we dive in
Suppose you ran a search engine over your logs the way you run one over web pages: ingest every log line, tokenize it, and build an inverted index from every word to every line that contains it. For a busy service emitting a million lines a second, that index alone, before you store a single compressed byte, dwarfs the logs themselves. The traditional Elasticsearch-style content index works, but its cost grows with the volume of words you ingest, and that cost is what broke the budgets of the teams who first tried to keep every log forever.
The problem that forced the distributed log aggregator into existence is that log content is huge and almost never the thing you search by first. A developer almost always begins a query by scope: which service, which namespace, which container, which pod. Only after narrowing to a stream do they grep the content. The insight, the one that Grafana Loki borrowed from Prometheus and inverted onto logs, is to index only that scope. A log stream is the set of lines that share the same labels, and the engine indexes the labels, never the words inside the line. The content is compressed into chunks and scanned in parallel only when a query asks for it.
The payoff is dramatic. The index shrinks by orders of magnitude because it holds label sets, not words. Chunks compress well because log lines within one stream are repetitive. And because the index is small, it lives entirely in cheap object storage alongside the chunks. The cost of storing a log for a year drops from a dollar-scale content index to cents-scale compressed storage. That is the trade this concept follows through every layer: pay nothing at ingest for content indexing, pay it lazily at query time by scanning the streams a label matcher selects.
But that trade reshapes the whole system. Because the index is labels and the content is chunks, the engine splits along a clean seam: a write path that validates labels, hashes them, and appends to in-memory chunks, and a read path that resolves label matchers to chunks and then scans. Mutability lives only in the ingester’s in-memory chunks; once a chunk is flushed to object storage it becomes immutable. Everything else, the decoupled engines, the quorum writes, the compactor, the per-tenant sharding, exists to preserve that seam.
Microservices topology and component interaction
The engine separates cleanly along the line where mutable memory becomes immutable storage. On the write side, the distributor and ingester own mutable chunks; on the read side, the querier, the index-gateway, and the compactor read immutable chunks and immutable index files. Nothing crosses back.
The distributor is the stateless gatekeeper of the write path. It receives a batch of streams, validates each label set, rate-limits per tenant, sorts the labels so the same set always hashes the same way, and forwards each stream to a replication factor of ingesters determined by a consistent hash ring. It waits for a quorum of acknowledgements before answering the client, so a write survives the loss of a minority of ingesters. Being stateless, the distributor scales independently to absorb traffic spikes and to shield the ingesters from denial of service.
The ingester is the only service that mutates data. It receives a stream, finds or creates the in-memory chunk for that tenant and label set, appends the lines, and periodically flushes a sealed, compressed chunk to object storage. Each accepted write is first recorded to a write-ahead log on local disk, so a crash before flush is recoverable by replay. The querier fans out for recent data to the ingesters and for historical data to the backing store, then deduplicates the replicated copies. The index-gateway is a dedicated, cache-like service over the index files: queriers ask it for the chunk references a label matcher resolves, so the heavy index work is isolated from the write path. The compactor runs as a singleton and merges the many per-day index files into one per tenant, while enforcing retention.
Decoupled read-write engines
The decision to run these as separately scalable engines is what lets the system absorb a write spike and a query spike at the same time. Logs are brutally write-heavy during incidents, and ironically that is exactly when operators run the most queries. If a single process served both paths, a flood of ingestion would starve the queries that are trying to diagnose the flood. Decoupling means the write engine (distributor, ingester) and the read engine (query-frontend, querier, index-gateway) can be sized for their own load curves.
Write path and read path lifecycles
A write reaches the distributor already batched by an agent. The distributor sorts and validates the labels, hashes the tenant plus label set to locate ingesters on the ring, sends the stream to a replication factor of them, and waits for a quorum to acknowledge. Each ingester appends the write to its WAL, adds the lines to the matching in-memory chunk, and acknowledges. When a chunk fills, sits idle too long, or a flush is forced, the ingester seals and compresses it and uploads it to object storage, where its content-addressed name lets replicas deduplicate.
A read arrives at the query-frontend, which splits a large range into smaller sub-queries and queues them fairly across tenants. A querier pulls a sub-query, asks the index-gateway for the chunk references that the label matcher resolves, fetches those chunks from object storage (and from ingesters for the in-memory tail), scans the decompressed lines against the content filter, deduplicates entries that share a timestamp, label set, and message, and returns its partial result. The frontend merges the partials into the final answer.
Mutability: append-only chunks and immutable blocks
The mutability boundary is absolute and is the property the whole topology protects. An in-memory chunk is append-only and mutable: new lines extend it. The moment it is flushed, it becomes an immutable object in storage that no writer can touch. The index files are similarly immutable once shipped: the compactor never edits them in place, it produces a new merged index and lets the old one expire. This is why a slow historical query cannot block an incoming write, and why a write burst cannot corrupt a running scan. Mutability is confined to the small, short-lived in-memory chunk; durability and concurrency are served by everything being immutable beyond it.
Concrete API schemas
The write contract carries the tenant, the stream identity (its label set), and the log entries; the query contract carries a time range, a label matcher, and a limit. The entries carry nanosecond timestamps, because logs from different sources arrive at different resolutions and a nanosecond field never truncates.
The separation between a batched Push RPC and a streaming QueryRange RPC is deliberate. Writes are cheapest when coalesced into one tenant-scoped batch, while reads are most responsive when the first matching lines stream back before the full scan ends. The selector narrows which streams the index must resolve; the filter runs later, against the decompressed content, which is exactly where the no-content-index trade is paid.
Low-level data structures and storage schema
In-memory label hashing, inverted index, and stream map
Inside the ingester, three structures hold the live data. The first is a map from the hash of a label set to its stream object. Because the distributor sorts the labels before hashing, the same label set always produces the same key, and that key is also what the consistent hash ring uses to place the stream on an ingester.
The hash folds the tenant id and the sorted label pairs into a fixed-width digest, the value the ring is walked against.
The second structure is the stream object itself: it holds the label set, the time bounds, and a list of chunks. The third is the in-memory chunk, an append-only buffer of compressed log lines plus a small per-block index of timestamps, so a range scan can skip blocks outside the window. The on-disk and in-object-store inverted index that lets a query find streams by label lives behind the index-gateway, but conceptually it is a postings structure: each label name-value pair maps to the list of streams (and thus chunks) carrying it.
A query like “app equals api” becomes a postings-list lookup, an intersection across multiple matchers, and then a fetch of the chunks those streams point to. Notice the asymmetry with a content index: this index resolves only label selectors, so the cost scales with the number of distinct label sets, never with the number of words ingested.
On-disk chunk files and the index
Once an ingester flushes, the in-memory chunk becomes an on-disk chunk file in object storage. A chunk is a self-describing container: a small header records a magic number, a version, and an encoding byte (the compression scheme, commonly Snappy or Gzip or Zstd). Below the header sit the compressed blocks of entries, and a metadata section records the per-block time bounds so a scan can skip whole blocks outside the query range.
The index, shipped and served by the index-gateway, maps label matchers to the chunks that hold matching streams. The recommended index is the Prometheus TSDB format: a symbol table that stores each distinct label string once, a series section mapping each stream to its labels and chunk references, and postings lists that map each label name-value pair to the sorted stream ids carrying it. Because the index is compact, it resides entirely in object storage, downloaded by the index-gateway and refreshed on a schedule.
Compaction: merging, dedup, and retention
Chunks and index files accumulate. The compactor’s job is to merge the many per-day index files an ingester ships into a single index file per tenant per day, so that an index lookup does not have to fan across dozens of small files. It runs as a singleton and works idempotently: it downloads the index files for a day, merges them into one, uploads it, and removes the old files.
Retention is enforced by the same compactor. For each day it traverses the merged index, finds the streams past their retention period, removes their chunk references from the index, writes those references to a marker file, and uploads the trimmed index. The chunks themselves are not deleted immediately; a sweeper deletes them after a configurable delay, which gives the index-gateways time to refresh their copies before the chunks vanish and leaves a window to cancel a mistaken retention rule. This is why deletion is a metadata operation that rewrites the index, not a rewrite of the chunks.
The two-stage shape, merge then trim, is what keeps compaction safe to repeat: merging only reorganizes index files without touching chunks, and trimming only rewrites the index, deferring the destructive chunk deletion to a delayed sweeper that runs after every reader has the new index.
Edge cases and chaos engineering
High-cardinality label explosion
The label-driven design is cheap until a label has unbounded values. A label like a request id or an ip address turns one stream into millions, because every distinct value is a new stream. The cost is paid in three places at once: the index balloons with new stream entries and postings, the ingester flushes thousands of tiny chunks (one per stream) to object storage instead of a few large ones, and every query that touches the high-cardinality label fans across all those streams. The defense begins at admission with a per-tenant cap on streams and labels, but the deeper discipline is to keep labels low-cardinality by nature: namespace, service, cluster, not user id. High-cardinality data that must be searchable belongs in structured metadata, which rides with each entry without entering the index.
Crash recovery and WAL or chunk replay
An ingester crash mid-chunk leaves in-memory lines unwritten and the WAL ahead of them. On restart the ingester replays its WAL in order, rebuilding each in-memory chunk from the records up to the last acknowledged write. Because every accepted write was appended to the WAL before acknowledgement, replay restores the tail exactly; nothing the distributor confirmed is lost. The WAL trades a little durability for availability: if a segment is corrupt the ingester recovers what it can and still starts rather than refusing to boot, and if the WAL disk fills, the ingester keeps accepting writes but stops logging them, accepting that those writes cannot survive a restart. A backpressure mechanism flushes to storage during replay when the WAL would exceed memory, so a WAL larger than RAM can still be replayed.
Multi-tenant isolation and per-tenant sharding
In a shared cluster every request carries a tenant id, and that id is the first key of isolation. All data, in memory and in object storage, is partitioned by tenant: the tenant id is part of the stream hash, part of the chunk’s content-addressed name, and part of the index file path. Rate limits, stream caps, and retention are all per-tenant. The index is sharded per tenant too, so that no single index file must hold every tenant’s streams, and the index-gateway can run in ring mode, distributing each tenant’s index across a set of gateway instances by hash. The result is that one tenant’s cardinality explosion cannot balloon another tenant’s index, and one tenant’s heavy queries cannot saturate another tenant’s index-gateway shard.
System hardening and programming realities
Garbage collection and heap pressure
The write path is a parsing and allocation hotspot. Every incoming batch decodes protobuf entries, parses label sets, and appends line bytes. If the ingester allocated a fresh buffer per line, the garbage collector would spend more time than the append. The remedy is fixed-capacity, recycled structures: a chunk is allocated once per stream and filled by index, sealed when full, and a new chunk takes its place. Label parsing reuses byte buffers, and the stream map is sharded to keep lock contention low under concurrent append. The steady-state allocation on the write path is engineered to be near zero, because the only thing worse than ingesting a million lines a second is collecting the garbage they produce.
Batching and zero-copy reads
On the read path the cost is decompression and scanning. Two techniques keep it bounded. First, batching: the query-frontend splits a large range into sub-queries sized so each shard processes a bounded amount of data, and it caches negative results (empty ranges) so a repeated query over a mostly-empty window short-circuits. Second, zero-copy reads: once a chunk is decompressed into a buffer, the scanner iterates it by offset without copying line bytes into intermediate objects, and the index-gateway serves chunk references from memory-mapped index files. Because flushed chunks and index files are immutable, a mapping never sees a torn read, and the page cache can be trusted as the source of truth.
Architectural evolution: 10x, 100x, 1000x
At 10x: single index lock contention
The first saturation is the single ingester’s structures under load. As the stream count grows, the lock on the stream map contends, and the in-memory index that resolves labels for the tail slows every query. The 10x response is sharding the stream map (one lock per shard), recycling chunk buffers to remove allocation pressure, and isolating the live index behind a cache so the read path stops contending with the append path. Compaction runs more frequently to keep the in-memory tail small. At this scale the system is still one logical engine; the fixes are internal.
At 100x: decoupled engines, bucket index, query splitting
At 100x, the coupling between write and read becomes the limit. A single engine cannot serve a write storm and a query storm at once. The response is to decouple into independently scalable read and write engines: distributors and ingesters on one side, query-frontends, queriers, and index-gateways on the other. The object-storage bucket itself becomes a bottleneck when every query lists it, so a bucket index records which chunks and index files exist and their time ranges, and queriers consult it instead of listing. Large queries are split into dynamically sized sub-queries that aim to process a bounded amount of data per shard, so a multi-day scan becomes many small parallel scans rather than one out-of-memory query. The compactor, now a singleton, merges index files per day to keep lookups efficient.
At 1000x: cells, async fabric, decoupled ingest-storage
At 1000x, the coupling between an ingester and its in-memory state is the limit. An ingester that owns both the WAL and the chunks must be replicated for durability, which wastes memory when the same tail lives in three heads, and a single global ring becomes a coordination bottleneck. The macro-shift is decoupled ingest-storage: writes land first in a durable, partitioned ingest log, and the ingester becomes a stateless consumer that builds chunks from the log and ships them to storage. Durability now lives in the log, not in replicated heads, so any ingester can rebuild by replay and scale freely. Beyond that the architecture moves to cells, each owning a slice of tenants with its own log, bucket, index, and ring, coordinated by an eventual async fabric rather than a single synchronous ring.
Notice what the log removes: the ingester no longer needs a replicated head for durability, so three in-memory copies of the recent tail collapse to one, and a failed ingester is rebuilt by replay rather than by copying a peer’s state.
How Grafana Loki does it
This design is the shape of Grafana Loki. Loki is a horizontally-scalable, multi-tenant log aggregation system that indexes only the set of labels for each log stream, never the content of the lines, and stores the compressed lines as chunks in object storage. That single decision, to make labels the index and content the payload, is what makes Loki cheaper to operate than a content-indexing aggregator, and every component exists to preserve the read-write seam it creates.
The write path is the distributor and the ingester. The distributor is stateless: it validates streams, rate-limits per tenant, sorts the labels so they hash deterministically, and uses a consistent hash ring over the tenant id plus label set to send each stream to a replication factor of ingesters. It waits for a quorum, defined as floor(replication factor divided by two) plus one, before acknowledging, so a write survives the loss of a minority. The ingester appends each accepted write to a write-ahead log, builds the stream into in-memory chunks, and flushes sealed compressed chunks to object storage; a chunk is compressed and marked read-only when it reaches capacity, is idle too long, or a flush occurs.
The read path is the query-frontend, the querier, and the index-gateway. The query-frontend splits large queries into smaller parallel sub-queries, queues them fairly across tenants, and caches both metric results and negative (empty-range) log results. The querier queries all ingesters for the in-memory tail before falling back to the backing store, and deduplicates entries that share a nanosecond timestamp, label set, and message. The index-gateway serves the metadata queries that resolve label matchers to chunk references, and can run in simple mode (all tenants on every instance) or ring mode (sharded per tenant). The compactor runs as a singleton, compacts the many per-day index files into one per tenant per day, and enforces retention by trimming chunk references into marker files and letting a sweeper delete the chunks after a configurable delay.
The index is the piece that has evolved most. Loki originally used a BoltDB-backed index shipped to object storage (boltdb-shipper, now deprecated). Since Loki v2.8 the recommended index is TSDB, borrowed from Prometheus, which stores the size in kilobytes and the line count of each chunk so the query-frontend can plan shards. TSDB performs dynamic query sharding, aiming to process roughly 300 to 600 megabytes per query shard, which means a large query becomes many smaller queries rather than one. Together these components are the concrete instance of the principle that a log aggregator is built to index only labels and preserve immutability, because those are the two properties that keep a write-heavy, high-volume log workload cheap and safely concurrent.
Mastery Questions
Question
Why does a distributed log aggregator index only labels and never log content, and what does that trade cost at query time?
Answer
Because log content is huge and almost never the first thing a query narrows by; a content index would cost more than the logs themselves. Indexing only labels shrinks the index by orders of magnitude and lets chunks live in cheap object storage. The cost is paid lazily at query time: after the label matcher selects streams, the engine must scan (decompress and grep) the chunk content to apply any content filter, instead of looking it up in an index.
Sources & evidence9 claims · 7 cited
All nine mechanism and operational claims are backed by fetched primary Grafana Loki docs (architecture, components, single-store TSDB index, WAL, retention, labels). The 10x/100x/1000x evolution is internal-reasoning extrapolation, not asserted as sourced fact.
- High-cardinality labels such as a request id or ip address cause the index to balloon with new stream entries, the ingester to flush thousands of tiny chunks, and queries to fan across all those streams, which is why labels should be low-cardinality and high-cardinality searchable data should use structured metadata.verified
- A Loki chunk file has a header with a magic number, version byte, and encoding byte naming the compression scheme, followed by compressed blocks of entries each with a checksum, and a metadata section recording per-block entry counts and min and max nanosecond timestamps.verified
- The compactor runs as a singleton, merges multiple per-day index files into a single per-tenant index per day, and enforces retention by trimming expired chunk references into a marker file and deleting the chunks asynchronously via a sweeper after a configurable delay so index-gateways can refresh.verified
- The ingester appends each accepted write to a write-ahead log, builds the stream into in-memory chunks, and seals and compresses a chunk as read-only when it reaches capacity, is idle too long, or a flush occurs, before uploading it to object storage.verified
- A distributed log aggregator indexes only the set of labels for each log stream and never the content of the log lines, compressing the lines into chunks stored in object storage.verified
- The querier queries all ingesters for in-memory data before falling back to the backing store, and deduplicates entries that share a nanosecond timestamp, label set, and log message to resolve replication duplicates.verified
- Since Loki v2.8 the recommended index is the Prometheus TSDB format stored in object storage, replacing the deprecated BoltDB shipper index, and it performs dynamic query sharding aiming to process roughly 300 to 600 megabytes per query shard.verified
- The write-ahead log records acknowledged writes to local disk so they survive a crash and are replayed on restart, but trades durability for availability: on a corrupt WAL it recovers what it can and still starts, and on a full WAL disk it keeps accepting writes but stops logging them so those writes cannot survive a restart.verified
- The distributor hashes the tenant id plus sorted label set to find ingesters via a consistent hash ring, forwards each stream to a replication factor of ingesters, and waits for a quorum of floor(rf/2)+1 acknowledgements before responding to the client.verified
Cited sources
- Understand labels · Grafana Labs
- Loki architecture · Grafana Labs
- Log retention · Grafana Labs
- Loki components · Grafana Labs
- Loki overview · Grafana Labs
- Single Store TSDB (tsdb) · Grafana Labs
- Write Ahead Log · Grafana Labs