High-Throughput Ingestion Pipeline

Learning outcomes

A telemetry pipeline fails the moment it tries to be polite. When ten thousand agents all flush their buffers at once, the system in front of them either absorbs the burst or it falls over, and there is no middle ground. This concept teaches the ingestion front of a petabyte-scale, multi-tenant observability platform: the boundary where untrusted, bursty writes become durable, sharded, rate-isolated data.

After studying this page, you can:

• Map the write-path microservices (distributor, ingester, hash ring) and explain how the read path is decoupled from them by immutable storage.
• Specify the gRPC push contract, including tenant identity, nanosecond timestamps, and label structures.
• Reason about ring buffers, persistent WAL-backed queues, and the bandwidth-delay product that governs backpressure.
• Diagnose high-cardinality bursts, mid-transaction ingester crashes, and noisy-tenant starvation.
• Apply allocation discipline, batching, and zero-copy techniques to keep garbage collection pauses off the hot path.
• Trace how the topology must change at 10x, 100x, and 1000x scale, ending in a cell-based, asynchronously federated fabric.
• Ground the design in how Grafana Mimir implements the distributor, ingester, and consistent hash ring.

Before we dive in

Imagine a billing platform that emits one counter per service per minute. A thousand services produce a gentle, predictable stream: a few hundred thousand samples a second, spread evenly. A naive HTTP server with a database behind it handles this without strain.

Now make it observability. Every microservice emits hundreds of distinct series, each tagged with high-cardinality labels like a request id or a user id. A deployment finishes, and ten thousand agents simultaneously decide to flush their buffers. A misconfigured job starts emitting a million new series a second. One tenant runs a loop that pushes the cardinality of the index through the roof.

The first failure is a story you already know: a slow consumer, a growing queue, an out-of-memory crash. But the real lesson of observability ingestion is that the workload is adversarial. The system is multi-tenant, so the load is shaped by other people’s mistakes, not yours. The pipeline cannot assume well-behaved writers, smooth rates, or bounded label sets. It must absorb bursts it did not cause, isolate tenants it does not control, and stay available when a single bad deploy tries to drown it.

That requirement forces a specific shape. The pipeline separates two concerns that a naive design conflates: admission, the decision of whether to accept a write, and durability, the act of making it survive a crash. Admission is stateless, cheap, and ruthless: it rate-limits, shards, and drops. Durability is stateful, careful, and slow: it writes a write-ahead log and flushes immutable blocks. The whole architecture exists to keep those two speeds apart, so that the ruthlessness of admission can protect the care of durability.

Microservices topology and component interaction

The write path is a chain of two services with a shared coordination structure between them. The read path is a different chain that touches the same storage but never the same in-flight data.

The distributor is the stateless entry point. It validates the request, enforces per-tenant rate limits, then decides which ingesters own each incoming series. It owns no data; if every distributor dies and restarts, nothing is lost, because durability lives in the ingesters. The ingester is the stateful core: it holds the most recent samples in memory, persists every accepted write to a write-ahead log first, and periodically compacts its memory into immutable blocks that it uploads to object storage.

Between them sits the consistent hash ring, a shared map of which ingester owns which shard of the series space. The distributor does not talk to a coordinator on the hot path; it reads the ring it has cached locally and hashes each series to find its owners.

Network protocols at each boundary

Every boundary in this design is a deliberate choice, not a default.

• Agent to distributor: gRPC over HTTP/2, Snappy-compressed Protocol Buffers. HTTP/2 multiplexing lets one TCP connection carry many concurrent streams, so an agent flushing a large batch does not pay TCP-handshake latency per series. Snappy is chosen over higher-ratio codecs like Zstandard because its compression and decompression cost per byte is low enough to stay off the hot path.
• Distributor to ingester: the same gRPC channel, but the critical property here is backpressure. The distributor streams writes to an ingester, and if that ingester’s WAL is saturating its disk, the ingester must be able to slow the distributor down rather than buffer the overflow in memory.
• Ingester to object storage: HTTPS with multipart PUT. This boundary is async and retries are cheap, because object storage is the slowest, most elastic sink in the system.

The read path is deliberately decoupled. A querier asking for recent data contacts ingesters directly, but for historical data it goes through the store-gateway, a cache over object storage. Neither read component ever touches the write queue, so a slow analytical query cannot block an incoming write, and a write burst cannot starve a query.

The write and read lifecycle, step by step

A write request travels a precise path. First, the agent batches its samples and opens a single gRPC stream to a distributor. The distributor extracts the tenant id from a request header and checks that tenant’s rate limit. If the limit is exceeded, the distributor rejects the batch with an HTTP 429 and discards it immediately; the agent is expected to back off and retry. If admitted, the distributor computes the hash of each series’ full label set and maps it onto the hash ring to find the N ingesters responsible for it, where N is the replication factor.

The distributor then forwards the write to all N ingesters in parallel. Each ingester appends the write to its write-ahead log on disk, and only once the log is fsynced does it add the sample to its in-memory series and acknowledge the distributor. The distributor waits for a quorum of acknowledgements (typically a majority, not all N) and returns success to the agent. If an ingester is slow or down, the distributor routes to a replacement the ring names, so a single failed ingester does not stall the write.

A read request travels a different path. The querier parses the query, asks the hash ring which ingesters currently own the matching series for recent data, and asks the store-gateway for historical blocks. It fans out, gathers partial results, merges them, and returns. The write queue is never on this path, which is why a burst of writes and a burst of reads do not collide.

Concrete API schemas

The push contract is the most-exercised interface in the system, called millions of times a second. It is specified in Protocol Buffers v3 so that field additions remain backward compatible and so the wire format stays compact and fast to decode.

Every metadata field earns its place. The tenant id drives rate limiting and sharding. The received timestamp lets the ingester detect and reorder out-of-order samples. The label set is the identity of the series, and its hash is the input to the ring lookup. A response that merely said “ok” or “error” would force every agent to retry blindly on throttle, amplifying the burst that caused the throttle in the first place.

Low-level data structures and buffering

The pipeline’s behavior under load is decided by three structures: the per-tenant rate limiter, the in-memory write buffer, and the write-ahead log on disk.

The rate limiter as a token bucket

Per-tenant admission is a token bucket, not a fixed window, because a token bucket forgives bursts up to a ceiling and then enforces a steady rate. Each tenant has a capacity of tokens that refills at a fixed rate; each incoming sample costs a token. When the bucket is empty, the request is rejected, not queued.

The bucket is evaluated lazily: rather than a background thread refilling it, the limiter computes the elapsed time since the last request and adds the accrued tokens on read. That makes the limiter lock-light and correct under concurrency. The crucial property is isolation: each tenant has its own bucket, so one tenant exhausting its tokens has no effect on any other tenant’s bucket.

The ingester’s memory: a sharded map plus a ring of chunks

Inside an ingester, the recently-arrived samples live in a structure that must support three operations cheaply: look up a series by its label set, append a sample in order, and evict compacted data to disk. The series are held in a concurrent map keyed by the hash of the label set, but each map entry points into a head chunk, a fixed-size append-only buffer per series. When a head chunk fills, it is sealed and a new one begins; sealed chunks are what compaction turns into block files.

The head chunk is a small ring buffer, typically a few kilobytes, sized to hold a few thousand samples. Its fixed size is what makes memory predictable: the ingester knows its worst-case memory is the number of series times the head chunk size, plus the sealed-but-uncompacted backlog. An unbounded map of samples, by contrast, would let one high-cardinality tenant exhaust the heap.

The write-ahead log and the bandwidth-delay product

The WAL is an append-only sequence of records on disk, segmented into fixed-size files. Every accepted write hits the WAL before it touches memory, so a crash that loses memory never loses a write the distributor already acknowledged. The cost of this safety is a disk fsync per batch, which is why the ingester batches writes aggressively before flushing.

Backpressure between distributor and ingester is not a feature the ingester adds; it inherits it from the transport. HTTP/2 flow control advertises a window per stream and per connection, and a slow ingester that stops reading lets its receive window drain, which forces the distributor’s HTTP/2 stack to pause its writes. The diameter of that window must cover the bandwidth-delay product of the link, or the stream will stall even when both ends are idle.

Edge cases and chaos engineering

A pipeline that works on steady load is not done. Three failure modes separate a toy ingestion front from a production one.

High-cardinality explosion

A label that takes unbounded values, like a request id or a user id, turns one logical series into millions. The first symptom is memory: each new series allocates a map entry and a head chunk, so a 100x surge in unique series inflates the ingester’s resident memory by roughly 100x. The second symptom, slower and more lethal, is index bloat: every new label pair is a new posting the inverted index must track, and queries that filter on those labels slow to a crawl.

The defense is layered. At admission, the distributor can reject series whose label cardinality exceeds a tenant-specific limit, returning a 400 that names the offending label so the operator can fix the source. At the ingester, a per-tenant cap on active series forces eviction of the oldest inactive series before memory is exhausted. At compaction, series that appeared briefly and never recurred can be merged or dropped. The key insight is that cardinality is an admission problem first: once an unbounded series reaches durable storage, every downstream stage pays for it, so the cheapest place to stop it is the distributor.

Crash recovery and ring rebalancing

Consider an ingester that crashes mid-transaction: it has appended a batch to its WAL, fsynced it, but not yet updated its in-memory map when it dies. On restart, the ingester replays its WAL segment by segment, rebuilding the in-memory series and head chunks exactly as they were at the last fsync. The distributor’s pending writes to this ingester were not acknowledged, so the distributor has already rerouted them to a replacement; the replay therefore restores data without duplicating it, because the replacement’s copy and the replayed copy converge during compaction.

The harder case is a crash with no restart, a node that leaves the ring permanently. The ring uses a gossip protocol so every distributor learns, within seconds, that the ingester is gone, and that its tokens have been redistributed to the survivors. Because each series was replicated to N ingesters, the loss of one leaves the data still present on the survivors, who now own it fully. Rebalancing moves ownership but not necessarily data: the new owners begin receiving fresh writes immediately, and only historical blocks that lived solely on the dead node would be unrecoverable, which is why the replication factor is set above the expected simultaneous-failure count.

The diagram’s key point is the gap between the fsync and the acknowledgement: the window in which a crash loses the in-memory copy but not the durable copy, and during which the distributor, having heard nothing, safely reroutes.

Multi-tenant isolation

A single tenant running a pathological query should not degrade ingestion for others, and neither should a single tenant’s write burst. The rate limiter isolates the write side by giving each tenant its own token bucket. The harder isolation is resource isolation: a tenant whose series cardinality is huge consumes more ingester memory, and without a cap its head chunks can crowd out everyone else’s. The remedy is per-tenant quotas on active series and on memory, enforced at the ingester, so that when a tenant exceeds its share the system sheds that tenant’s data rather than degrading globally.

System hardening and programming realities

The ingestion front is a system where the language runtime’s behavior is load-bearing. A garbage collection pause of a few hundred milliseconds, invisible in a request-response service, here means a stalled HTTP/2 stream, a drained receive window, and a cascading retry storm.

Memory and garbage collection

The dominant allocation pressure on the write path is the deserialization of incoming Protocol Buffer batches: each PushRequest allocates Label and Sample objects by the thousand. Left uncontrolled, this fills the young generation rapidly and triggers frequent collections, each of which stops the world long enough to cause backpressure spikes. The standard discipline is to pool buffers and reuse decoders, so a sustained batch rate does not produce a sustained allocation rate. The ingester’s series map and head chunks are pre-sized and append-only, so the steady-state allocation of the hot path approaches zero once the head chunks are warm.

Go, the language this family of systems is written in, lets the operator tune the garbage collector’s target percentage. Lowering it (making GC run more often but for shorter pauses) trades some CPU for lower tail latency, which is the right trade on a latency-sensitive write path. The signal to watch is not average latency but the p99, because the p99 is what the retrying agent experiences as a timeout.

Zero-copy and batching

Every byte that crosses a process boundary without being copied is throughput earned for free. Snappy decompression and Protobuf parsing can be done over a borrowed network buffer rather than a copied one. Object-storage uploads use multipart PUTs that stream directly from a memory-mapped block file, so a 100-megabyte block is uploaded without ever being copied into a userspace staging buffer. Batching is the other lever: the cost per sample of a WAL fsync is divided by the number of samples in the batch, so the ingester’s first job on receiving a batch is to coalesce it with everything else pending before it touches disk.

Architectural evolution: 10x, 100x, 1000x

A topology that is elegant at one million samples per second becomes a bottleneck at one hundred million. The evolution is forced by specific saturation points.

At 10x: the network and the lock

The first saturation is rarely CPU; it is the single TCP connection or the single shared lock. Distributors that held one gRPC channel per agent hit connection-tracking limits, and the series map’s global lock contends under concurrency. The 10x response is horizontal scaling of the stateless distributors (they can be added freely since they hold no data) and sharding the ingester’s series map into a fixed number of shards, each with its own lock. A basic cache topology appears: a query-frontend that memoizes identical queries, absorbing the read-side amplification that 10x traffic brings.

At 100x: the ring, the bucket, and the query

At 100x, the consistent hash ring itself becomes the limit. A ring with thousands of ingesters means gossip convergence slows, and the distributor’s ring cache grows large enough that lookups cost real CPU. The single object-storage bucket, naively shared, hits request-rate limits. The response is shuffle sharding, which assigns each tenant to a small subset of ingesters so a tenant’s blast radius is bounded and gossip stays local; bucket-based indexing, which replaces the assumption that a querier must list the bucket; and query splitting, which breaks a large range query into sub-queries fanned across queriers and merged. Multi-tenant isolation moves from rate limits into dedicated quotas per tenant per shard.

At 1000x: the cell

At 1000x, a single globally-shared ring and a single object-storage bucket are no longer tenable. The architecture shifts to cells: independent, self-contained deployments of the entire write-storage-read stack, each owning a slice of tenants. A regional edge proxy routes a write to the cell that owns its tenant, so no cell holds global state. Synchronous coordination, the gossip ring and the global bucket index, is replaced by an asynchronous message fabric that propagates tenant-to-cell mappings eventually rather than immediately. The system gives up global consistency in exchange for the only property that survives hyperscale: a failure in one cell cannot take down the rest.

The cell boundary is a fault boundary: a write that enters cell one cannot block a write in cell two, because they share no queue, no ring, and no synchronous path.

How Grafana Mimir does it

This design is not abstract; it is the shape Grafana Mimir takes. Mimir is a horizontally scalable, highly available, multi-tenant, long-term-storage time-series database for Prometheus and OpenMetrics data. Its write path is exactly the distributor-to-ingester chain described here.

The Mimir distributor is a stateless entry point that validates data, enforces per-tenant limits, then shards incoming series. Per-tenant rate limits are configured as a request rate and an ingestion rate, and the distributor drops or throttles with a 429 that carries a retry hint. The ingester stores the most recently ingested samples both in memory and on disk, compacting them into TSDB blocks every two hours before uploading to object storage. A write-ahead log persists every accepted write, so an ingester restart replays from the WAL and loses nothing the distributor had acknowledged.

The coordination layer is Mimir’s consistent hash ring. Mimir uses a distributed consistent hashing scheme over a 32-bit token space for sharding, replication, and service discovery. The distributor hashes each series’ label set and routes to the ingesters whose tokens own that region of the space. Membership is spread through gossip, so distributors learn of ingester joins and leaves without a central coordinator. The token strategy is spread-minimizing: rather than random tokens that cluster unevenly, tokens are generated to spread ownership evenly across the ring, which keeps the rebalancing cost of an ingester join or leave to a small fraction of the keyspace. This is the concrete instance of the general principle that admission must be ruthlessly stateless while durability must be carefully stateful, and that the hash ring is what lets them cooperate without coupling.

Mastery Questions