Real-Time Observability Backend

Learning outcomes

A dashboard that polls a query backend every second is not hard because the single query is slow, it is hard because a thousand dashboards polling at once turn a quiet read path into a sustained thundering herd. The real-time observability backend exists to replace that herd with one push: the data source tells the browser when a new sample exists, and the browser draws it. Building that push is a different problem from building a query, because the backend must hold a live connection open per viewer, fan one upstream stream out to many viewers, and stay fair when a viewer cannot keep up.

After studying this page, you can:

• Name the services on the live-data path (the ingester or pubsub broker tail, the streaming query-frontend, the SSE or WebSocket gateway, the query cache) and trace a sample from source to browser.
• Specify the proto3 contract for a streaming live-values RPC, and the JSON envelope a browser receives over SSE or WebSocket.
• Explain the fan-out tree, the per-stream ring buffer, query caching of identical live queries, and progressive partial rendering.
• Diagnose slow-consumer backpressure, subscriber crash and disconnect, cache staleness, and multi-tenant fan-out unfairness, and name their mitigations.
• Reason about garbage collection from per-subscriber buffers, zero-copy frame forwarding, memory bounds per subscriber, and write coalescing.
• Map the 10x, 100x, and 1000x evolution onto a real-time telemetry push architecture.
• Describe how Grafana Live pushes live data to dashboards and how datasource streaming works.

Before we dive in

Picture an operations team that wants a live CPU panel. The obvious implementation is to set the dashboard to auto-refresh every second. Each refresh issues the same range query to the metrics backend, and the backend dutifully resolves every series, scans the relevant chunks, evaluates the expression, and returns the same answer it returned one second ago, plus one new point. One panel at one refresh a second is trivial. But a dashboard with many panels refreshing every ten seconds already generates roughly six times the query load of the same dashboard refreshing every minute, and that load scales with the number of people who have the dashboard open. Five engineers open the same page, and the backend now answers the same question five times per refresh tick.

This is the thundering herd. The backend does work proportional to the number of viewers, not the amount of new data, because polling asks the question “what changed?” even when nothing changed. The query path, already built to handle heavy historical scans, is now being hammered with identical short-range questions every second by every open tab. Concurrent users issuing queries or causing panels to refresh become the primary driver of CPU and memory load, and the backend spends most of its effort re-answering questions it just answered.

The fix is to invert the flow. Instead of every browser asking the backend for the latest value on a timer, the backend holds one open path to each browser and pushes a new value the moment it exists. One new sample enters the ingester, and the backend fans it out to every viewer who asked for that series, exactly once regardless of how many viewers there are. That reframe, from polling many to pushing from one, is what a real-time observability backend is. Every component on the live-data path exists to make that one-to-many push cheap, bounded, and fair, and to keep a slow browser from dragging the rest down.

Microservices topology and component interaction

The live-data path is a pipeline whose front speaks the browser’s protocol and whose back tails the data source for new samples. Unlike a one-shot query path, which fetches a closed range and returns, this path holds state for as long as a viewer is watching, because a live subscription is a long-lived connection, not a request.

Four roles matter, each earned by the problem it solves.

The ingester tail (or pubsub broker subscription) is the source of new samples. It is the live counterpart of the historical leaf in a one-shot query. It either emits a stream of new samples itself, or the backend holds a long-lived subscription to a broker topic and translates each published message into a sample frame. Nothing downstream polls it; the tail pushes.

The streaming query-frontend is the coordinator. It owns the fan-out: it deduplicates identical live queries so that two browsers watching the same panel share one upstream subscription rather than opening two, it memoizes recent results in a query cache, and it merges or transforms frames before forwarding them. It holds the live state, which is what makes it the place where memory must be bounded.

The SSE or WebSocket gateway is the browser-facing edge. It speaks the client’s protocol and holds one persistent connection per viewer. It terminates the streaming transport, handles reconnection, and forwards each frame the frontend emits onto the right socket. Over WebSocket a single connection can multiplex many subscriptions; over SSE each event stream is one subscription, which is why SSE over HTTP/2, with its multiplexed streams, matters at scale.

The query cache sits in front of the fan-out. Before the frontend opens a new upstream subscription for a live query, it checks whether an identical one is already running, and serves new viewers from the existing stream. A hit means one fewer upstream subscription; a miss means the frontend opens the subscription and future identical queries join it.

The wires differ by direction. Between the gateway and the browser the protocol is WebSocket or SSE over HTTP/2, because those are the transports a browser can hold open and read pushed frames from. Between the frontend and the ingester the protocol is a server-internal gRPC stream, because both ends are trusted services and gRPC streaming carries framed messages efficiently over a single HTTP/2 connection.

The read lifecycle for a live subscription

A single live subscription moves through the pipeline in a fixed order: open, subscribe, share, push, close. This differs from a one-shot query, which is open, fetch, return, done, because a live subscription stays alive for the whole time the viewer is watching.

Each step earns its place. Open is the one-time cost of establishing the persistent transport; it is expensive, so the system does it once per viewer and reuses it for every panel. Subscribe is where the frontend decides whether to open a new upstream or join an existing one, which is the deduplication that breaks the thundering herd. Share is the moment the one-to-many fan-out begins, one upstream feeding N downstreams. Push is the steady state, each new sample traveling from ingester to every viewer in a single hop through the frontend. Close is the cleanup that prevents leaked subscriptions: when a viewer disconnects, its entry is removed from the fan-out, and only when the last viewer for a query leaves does the frontend close the upstream, so the ingester stops doing work nobody is watching.

Concrete API schemas

The live contract must carry everything the frontend needs to open one shared upstream and everything the gateway needs to forward a frame to a browser. The server-internal RPC carries the query that selects series and the live flag; the browser envelope carries the event type, the series identity, the timestamp, and the value.

The streaming return type is the whole point. A unary RPC would force the caller to re-ask for the next value; a stream lets the ingester push the next sample the moment it exists, which is what turns polling into push.

The browser does not speak protobuf. It receives a JSON envelope over SSE or WebSocket, carrying the event type, the series identity, the timestamp, and the value.

The envelope separates routing from payload. The outer type tells the gateway and browser what kind of frame this is; the inner data is the sample itself. Over SSE each envelope is one text block terminated by a pair of newlines; over WebSocket each envelope is one message frame, which is why both transports can carry the same payload with different framing.

Low-level data structures and execution

The fan-out tree

The core data structure is the fan-out tree: one upstream subscription feeds N downstream subscribers. Each node in the tree owns one upstream stream and a list of downstream channels. When the frontend receives a request for a live query, it hashes the query parameters and looks up the matching node. If one exists, the new viewer is appended to its downstream list and no new upstream is opened. If none exists, the frontend opens one upstream subscription and creates a node whose downstream list starts with this single viewer.

The tree is what makes cost independent of viewers. One ingester stream does the work; whether one browser or a thousand watch the same panel, the upstream cost is paid once. The downstream cost is only the work of copying each frame onto each viewer’s channel, which is cheap and bounded per subscriber.

Per-stream ring buffer and query caching

A viewer that joins a live stream late should not see an empty chart while it waits for the next sample. Each fan-out node keeps a per-stream ring buffer of the most recent samples, sized to cover a short window (the last minute or two). When a new downstream joins, the frontend replays the ring buffer contents first, so the late viewer sees recent history immediately, then continues with the live stream. The ring is a fixed-size circular buffer, so adding a new sample overwrites the oldest, and the memory cost per node is constant regardless of how long the stream has been running.

Query caching is the memoization layer that sits in front of the fan-out tree. The frontend hashes the live query parameters and treats that hash as the cache key. The first viewer opens the upstream and populates the entry; every subsequent identical query is a cache hit that joins the existing stream. This is deeper than a one-shot result cache, because a live cache entry is not a static answer but an open stream that keeps producing. Its lifetime is tied to subscribers: when the last downstream leaves, the entry is evicted and the upstream closes, so the cache never holds streams nobody is watching.

The total memory the live path can consume has a clear bound. Each subscriber adds the cost of its entry in the fan-out list plus its share of the per-node ring buffer replay.

Live memory per node is the number of downstream subscribers times the entry size, plus a fixed ring of K samples, so bounding subscribers per node and the ring size K bounds total live memory.

Because the ring term is constant per node, the only term that grows with scale is subscribers times the per-entry size, which is small. That is what lets the frontend hold many open live queries without unbounded growth.

Progressive rendering at the query-frontend

Some live queries are not pure pass-throughs. A live panel that shows the average CPU across a fleet must aggregate many series into one, and that aggregate must be recomputed as each new sample arrives. The query-frontend does this progressively: as partial series stream in from the ingester, it emits a partial result immediately, then refines it as more samples arrive. The browser sees the chart update incrementally rather than waiting for a complete recompute on every tick.

This progressive rendering is the live analogue of partial-result merging in a one-shot query. The frontend never blocks on a complete answer; it ships its best current result and improves it, which is what keeps the perceived latency of a complex live panel as low as a simple one.

Edge cases and chaos engineering

Slow consumer backpressure

The one-to-many fan-out has a failure mode the one-shot path does not: a browser that cannot keep up. A viewer on a slow phone, or one whose tab is backgrounded and throttled, receives frames slower than the ingester produces them. The WebSocket interface in the browser does not natively support backpressure, so when messages arrive faster than the application can process them, the connection fills device memory by buffering, becomes unresponsive through high CPU usage, or both. On the server side, a slow downstream means frames pile up in its channel buffer, and unbounded buffering is an out-of-memory waiting to happen.

The gateway must choose one of three responses, and the choice is a policy decision, not an accident. Drop discards the oldest undelivered frames for that subscriber, keeping the buffer bounded at the cost of a gap in that viewer’s chart. Sample-down coalesces multiple pending frames into one that carries the latest value, preserving recency while collapsing the backlog. Close terminates the slow connection outright, forcing the browser to reconnect and rejoin at the current point. Each trades freshness for memory safety; the right default for telemetry is usually drop or sample-down, because a live chart tolerates a missing point better than a dead server.

Subscriber crash and disconnect

A viewer whose browser crashes, loses network, or simply closes the tab does not send a polite goodbye. The gateway must detect the loss, either through the transport’s close handshake or through a missed ping or pong, and remove that subscriber from its fan-out node. The closing handshake in the WebSocket protocol exists precisely because the underlying TCP close is not always reliable through proxies, so the server cannot assume a silent socket is still alive. A leaked subscriber entry means the frontend keeps copying frames onto a channel nobody reads, a slow memory leak that grows with churn. Heartbeats (periodic pings) and a deadline on pong responses are what bound the detection time.

Cache staleness and multi-tenant fan-out fairness

The live cache entry is an open stream, not a snapshot, so staleness here is a different beast from a stale one-shot result. The risk is a node whose upstream has silently died (the ingester stopped pushing without closing) while downstream viewers keep receiving nothing, staring at a frozen chart. A watchdog that pushes synthetic heartbeats into each node, and closes and reopens any upstream that has not produced a sample within a deadline, keeps the cache honest.

Multi-tenant fairness matters because the fan-out tree is shared infrastructure. A single tenant running thousands of live panels could open thousands of upstream subscriptions and crowd out everyone else’s nodes. The fix mirrors the one-shot path: per-tenant limits on open live queries and on concurrent subscribers, plus fair scheduling when the gateway’s connection budget is contested. Without these caps, one noisy tenant freezes live panels for the whole fleet.

System hardening and programming realities

Garbage collection and per-subscriber buffers

Every fan-out copy of a frame is a short-lived object, and a high-frequency live stream produces them by the thousand per second. Allocating a new buffer for each downstream viewer on each sample guarantees a garbage-collection pause proportional to the subscriber count times the sample rate, which is exactly the stutter the user sees as chart jank. The discipline is to reuse. Each downstream channel owns a reusable byte slice, and the frontend serializes a frame into that slice once per subscriber and hands it to the gateway, rather than allocating fresh. The same principle bounds the ring buffer, which is a pre-allocated circular array of fixed-size slots, overwritten in place, so a long-running live stream produces zero per-sample allocations in steady state.

Zero-copy frame forwarding

The cheapest frame to forward is the one the server never copies. When the ingester hands the frontend a serialized frame, the ideal path forwards that exact byte slice to every gateway without re-serializing per viewer. The frontend reads the frame once, and writes references to it into each downstream channel. The gateway then writes the same immutable bytes onto each socket. Go’s model of cheap slice sharing makes this natural, provided nothing in the path mutates the slice. The moment any transform (an aggregation, a label rewrite) is needed, the frontend must copy, but it copies once into the per-subscriber reusable buffer, not once per frame across all subscribers. Zero-copy forwarding is what lets a node fan one frame out to thousands of viewers without the memory traffic scaling with viewers.

Bounding memory per subscriber and write coalescing

A live path that holds state per viewer must bound that state. Each subscriber entry carries its channel buffer, its replay cursor into the ring, and its connection metadata. A per-subscriber memory cap, enforced as a maximum channel buffer size, prevents one slow viewer from bloating a node. When the buffer fills, the drop or sample-down policy from backpressure applies, so the bound is enforced by design, not by hope.

Write coalescing is the complementary optimization on the upstream side. When many samples arrive in a burst (a scrape that returns a hundred series at once), serializing and forwarding each one individually multiplies gateway writes a hundredfold. Coalescing collects the burst into one frame carrying many samples, and the gateway writes once, letting the browser split the batch on arrival. Coalescing trades a small latency increase (the time to fill the batch) for a large reduction in write syscalls and per-frame overhead, which at high cardinality is the difference between a smooth stream and a saturated gateway.

Architectural evolution: 10x, 100x, 1000x

At 10x: a single gateway

At modest scale the live path is one process that tails the ingester, holds the fan-out nodes in memory, and serves the browsers directly over a single gateway. The query cache and the fan-out tree live in the same address space, and the ring buffers are small. The failure mode here is the connection ceiling: a single process can only hold so many open sockets before it hits file-descriptor or memory limits, and the first popular live dashboard shared across the team exposes it. The lesson learned is that holding a connection per viewer does not scale linearly on one box, and the persistent-connection budget, not the query cost, becomes the binding constraint.

At 100x: sharded fan-out tree and shared cache

Growth forces the split. The fan-out tree is sharded across several frontend instances by a hash of the query, so each node lives on exactly one frontend, and a viewer is routed to the right one. The query cache moves to a shared store (a Redis-like layer) so that the deduplication decision is consistent across frontends, and the SSE or WebSocket gateway becomes a separate pool that terminates browser connections and pulls frames from the right frontend. A shared Redis Pub/Sub layer also carries fan-out across the gateway pool, so a frame published by one frontend reaches a viewer held by any gateway. The new failure mode is cross-instance fairness and cache coherence, which is what drives the per-tenant caps and the shared cache.

At 1000x: edge POPs and regional fan-out

At the largest scale the binding constraint becomes geography and connection volume, not query cost. The architecture deploys edge points of presence (POPs) close to viewers, each terminating browser connections locally and subscribing back to a regional fan-out layer. The regional layer holds the sharded fan-out tree and the shared cache for a region, and an async fabric (a durable pubsub or log) carries samples between regions so a globally watched panel is fed from one logical upstream. The fan-out becomes hierarchical: ingester to regional frontend, regional frontend to edge POP, edge POP to browser. The throughline is unchanged: one upstream feeds many downstreams at every level, and every level bounds its per-viewer state, because the cost of holding a million open connections is what the design is really fighting.

How Grafana does it

Grafana implements this design as Grafana Live, a real-time messaging engine that pushes event data to a frontend as soon as an event occurs. Grafana Live sends data to clients over persistent WebSocket connections using a Pub/Sub model: the Grafana frontend subscribes to channels and receives whatever is published to them, and all subscriptions on a page are multiplexed inside a single WebSocket connection.

A Grafana Live channel is a string of three parts: scope, namespace, and path. For data source streaming the scope is ds, the namespace is the data source’s unique ID, and the path is a plugin-defined string. A data source channel looks like ds followed by the data source UID and a custom path. Channels are lightweight and ephemeral: they are created automatically when a user subscribes and removed as soon as the last user leaves, which is exactly the fan-out node lifecycle of the general design.

Datasource streaming works through backend data source plugins that implement a stream handler. The plugin’s RunStream method runs once per connection and sends data frames continuously until the channel closes, using a sender that emits each frame as a Grafana data frame of time and value fields. On the frontend, the data source calls the Grafana Live service to open a data stream addressed by scope, namespace (the data source UID), and path, where the path distinguishes channels so each distinct query creates its own stream. This is the concrete embodiment of one upstream feeding many downstreams: one running RunStream per channel, and every browser watching that channel receiving the frames it produces.

The scaling realities match the general design. Each persistent WebSocket connection costs about 50 KB of memory on the server, so a server with 1 GB of RAM is expected to handle about 20,000 connections at most, and Grafana Live defaults to a maximum of 100 simultaneous connections per server instance, tunable upward only when the operating system and infrastructure allow more. In a high-availability setup of several Grafana instances behind a load balancer, the in-memory Pub/Sub hub is not shared, so Grafana Live has an HA engine backed by Redis: it keeps its state in Redis and uses Redis Pub/Sub to deliver messages to all subscribers across all Grafana server nodes, which is the shared-cache-and-cross-instance-fan-out pattern the general architecture reaches for at 100x.

Mastery Questions

Question

A team opens the same live dashboard on five laptops and the metrics backend's CPU spikes, even though almost no new data is arriving. Name the failure and the single design change that fixes it.

Question

Two browsers open the same live panel. What data structure makes the ingester do the work only once, and what happens when the second browser joins late?

Question

A viewer on a slow phone cannot keep up with the sample rate and frames pile up in its channel. Name the three gateway policies and which is usually the right default for a live chart.

Question

Why does the live query cache evict an entry only when the last subscriber leaves, and what watchdog keeps a frozen upstream from serving stale silence?

Question

A high-frequency stream causes garbage-collection jank on the frontend. Explain the per-subscriber buffer discipline that removes per-sample allocations in steady state.

Question

Map Grafana Live onto the general real-time backend design: what is the channel, what runs the upstream, and what makes fan-out work across multiple Grafana instances?

Answer

This is the thundering herd of polling: each browser re-asks the same query on a refresh timer, so backend work scales with viewer count, not new data. The fix is to invert the flow and push from one source to many, holding one open transport per viewer and fanning each new sample out to every subscriber exactly once.

Answer

A fan-out tree node keyed by the hashed query holds one upstream stream and a list of downstream subscribers, so the second browser joins the existing node and no new upstream opens. The node keeps a per-stream ring buffer of recent samples, so the late viewer first receives a replay of the ring for immediate history, then continues with the live stream.

Answer

Drop discards the oldest undelivered frames for that subscriber; sample-down coalesces pending frames into one carrying the latest value; close terminates the connection and forces a reconnect. The default for telemetry is usually drop or sample-down, because a live chart tolerates a missing point better than a server that runs out of memory from unbounded buffering.

Answer

A live cache entry is an open stream, not a snapshot, so it must stay alive as long as anyone is watching and close the upstream the moment the last viewer leaves, otherwise the frontend leaks work on streams nobody reads. A watchdog pushes synthetic heartbeats into each node and closes and reopens any upstream that has not produced a real sample within a deadline, so a silently dead ingester does not freeze the chart.

Answer

Each downstream channel owns a reusable byte slice, and the frontend serializes a frame into that slice once per subscriber rather than allocating fresh, while the ring buffer is a pre-allocated circular array of fixed slots overwritten in place. Zero-copy forwarding shares one immutable serialized slice across all viewers, copying only when a transform is needed, so steady-state allocation is zero and the GC pressure stops scaling with subscriber count times sample rate.

Answer

A Grafana Live channel is scope/namespace/path (for datasource streaming, scope $ds$, namespace the data source UID), acting as the fan-out node key and created or destroyed with its subscribers. The upstream is the backend plugin's RunStream, which runs once per channel and sends data frames until it closes. Across instances the in-memory hub is replaced by a Redis HA engine that keeps state in Redis and uses Redis Pub/Sub to deliver each frame to subscribers on every node, the shared-cache and cross-instance fan-out pattern.

Question

A team opens the same live dashboard on five laptops and the metrics backend's CPU spikes, even though almost no new data is arriving. Name the failure and the single design change that fixes it.

Answer

This is the thundering herd of polling: each browser re-asks the same query on a refresh timer, so backend work scales with viewer count, not new data. The fix is to invert the flow and push from one source to many, holding one open transport per viewer and fanning each new sample out to every subscriber exactly once.

Prev1 / 6Next