On this path · Control Plane 2. Global Synthetic Monitoring Infrastructure
Global Synthetic Monitoring Infrastructure
Worldwide probe orchestration, network-partition resilience, and real-time SLA/SLO aggregation.
Learning outcomes
Your own region cannot tell you that a user three thousand miles away is broken. The failure is invisible to every metric you collect from inside your own datacenter, because the request never reached your servers, or it reached them late, or it reached them with a certificate a regional resolver decided to distrust. This concept teaches the control plane that closes that blind spot: a globally distributed fleet of probes that actively tests your endpoints from the outside, on a schedule, and feeds the results into the same SLA and SLO machinery that your internal metrics use.
After studying this page, you can:
- Map the synthetic-monitoring control plane (scheduler, probes, ingestion path, SLA/SLO aggregation) and explain why scheduling is decoupled from probing.
- Specify the gRPC check-dispatch and result-push contracts, including target, region, interval, timeout, and success criteria.
- Reason about per-region schedule queues, consistent-hash probe assignment, and rolling error-budget burn windows.
- Diagnose network partitions between scheduler and probes, and between probes and ingestion, plus multi-tenant probe fairness and clock skew.
- Apply allocation discipline and backpressure to keep result parsing and windowed counters off the garbage-collection hot path.
- Trace how the topology must change at 10x, 100x, and 1000x scale, ending in cell-based, edge-local probe autonomy.
- Ground the design in how Grafana Synthetic Monitoring runs probes, checks, the blackbox exporter, and feeds results into Grafana Cloud.
Before we dive in
A team runs a checkout service out of a single region and watches its dashboards religiously. Latency is flat, error rate is zero, CPU is calm. Then a customer in another continent opens the page and it hangs for eight seconds, or never loads at all. The team sees nothing. Their dashboards are honest, but they answer the wrong question: they report what the service experiences, not what the user experiences. Between the user and the service sit a DNS resolver, a CDN edge, a certificate chain, a transit provider, and a regional filter that may be silently dropping traffic. None of those belong to the team, and none of them emit metrics into the team’s database.
The only way to see that gap is to become the user, on purpose, from the place the user sits, on a clock. That is what forced active, external probing into existence. Instead of waiting for the service to report its own distress, you station lightweight agents around the world whose entire job is to impersonate a user, perform one complete interaction against a target, and report whether it worked and how long it took. Each run is a single emulated iteration from one location, deliberately lightweight and repeatable, so a failure means a real user is failing too.
That requirement reshapes the architecture in a specific way. It forces a split between two planes that a naive design would conflate. The control plane decides what to test, from where, and how often: it holds the check definitions and the schedule. The data plane does the actual testing: it is the fleet of probes, spread across regions, each running its assigned checks independently. The scheduler never performs a probe, and a probe never decides the global schedule. This separation is what lets the system keep probing when a region is cut off, because a probe that can no longer reach the scheduler can still reach the target, and a probe that can no longer reach the ingestion path can still keep testing and forward later. The whole design exists to make the act of observation survive the same network failures it is trying to observe.
Microservices topology and component interaction
The control plane is a chain of four roles, and the data plane is a fleet that fans out beneath it. The two are connected by messages, not by shared memory, which is the property that lets them fail independently.
The scheduler is the stateful brain of the control plane. It holds the set of check definitions: for each check, a target endpoint, a set of probe regions, an interval, a timeout, and a success criteria. On each interval tick it works out which probes owe a run and dispatches the relevant definitions. It owns the schedule but performs no probing itself. The probes are the data plane: lightweight blackbox-style checkers, each pinned to a region, each executing its assigned checks independently and on its own clock. Crucially, every probe selected for a check runs that check at every interval; probes do not take turns. A check assigned to five probes on a one-minute interval produces five runs per minute, not one, because each run is the answer to a separate question: is the target healthy from here.
The ingestion path is the conduit results flow back through. Each probe, after a run, pushes an outcome record (timestamp, success, latency, error) toward the ingestion path, which validates, deduplicates by check and probe, and forwards the stream to the SLA/SLO aggregation engine. That engine turns a river of per-run booleans into the things an operator actually alerts on: success rate over a window, error-budget burn, and latency percentiles. The decoupling matters most here. Because scheduling, probing, and aggregation are separate, a slow aggregation engine cannot stall a probe, and a probe that falls behind cannot stall the scheduler.
A probe that holds the schedule would, on losing contact with its peers, keep guessing at a schedule it can no longer reconcile. A scheduler that performed its own probing would, on a region going dark, lose both the schedule and the only observer that could have reported the darkness. Splitting them means the observer outlives the failure it observes.
Protocols at each boundary
Every boundary in this design is a deliberate choice.
- Scheduler to probe: gRPC over HTTP/2, carrying check definitions. The probe holds a long-lived stream to its regional scheduler and receives incremental dispatch updates, so adding or changing a check does not require a poll loop or a full re-fetch. HTTP/2 multiplexing lets many definition updates share one connection.
- Probe to target: the native protocol of the check itself (HTTP, HTTPS, DNS, TCP, ICMP). This is the blackbox boundary: the probe knows nothing about the target’s internals, only whether the interaction it was told to perform succeeded within the timeout.
- Probe to ingestion path: OTLP over gRPC. Results are pushed, not scraped, because a probe behind a restrictive egress or in a region that blocks inbound connections cannot be polled by the ingestion path. OTLP gives a typed, batched, backpressure-aware channel that fits the same metrics pipeline the rest of the platform already speaks.
The success signal itself comes from the blackbox checker. After performing the configured interaction, the checker emits a single decisive boolean, the equivalent of the blackbox exporter’s probe success metric, alongside a duration. The ingestion path treats that boolean as the atomic unit of SLA truth: every downstream window, burn rate, and alert is an aggregation of these booleans, never a re-derivation from logs.
Partition resilience: probing through a split
A synthetic monitor that stops probing during a network event is worse than useless, because the event is exactly what it existed to catch. The design therefore assumes partitions are normal, not exceptional. When the scheduler loses contact with a probe region, that region’s probes keep probing: each holds its last-received check definitions and continues to run them on their intervals against the target. The probes are observers first and clients of the scheduler second. When the ingestion path loses contact with the probes, the probes buffer their results locally and forward them on reconnection, so the aggregation engine eventually sees a complete record rather than a silent gap. These two fallbacks are examined in detail under Edge cases.
Concrete API schemas
The dispatch and result contracts are the most-exercised interfaces between the control and data planes. They are specified in Protocol Buffers v3 so that adding a field stays backward compatible and the wire format stays compact and fast to decode on a high-frequency results path.
Every field earns its place. The tenant id drives fairness and isolation on shared probes. The region lets the scheduler place work where the observation must originate. The interval and timeout bound a probe’s worst-case work per check, which is what keeps a single tenant from saturating a probe. The success boolean is the one bit the whole SLA machinery reduces to; without it, the system would have to reconstruct success from logs, which is slow and ambiguous. The retry hint on the ack is what prevents a recovered probe from stampeding the ingestion path with a backlog all at once.
Low-level data structures and execution
The control plane’s behavior under load is decided by three structures: the schedule queue that decides when a check is due, the assignment that decides which probe runs it, and the rolling window that turns results into an error-budget burn rate.
The schedule queue: per-region, per-interval ticks
The scheduler does not keep a list of checks and wake each one on a timer. Instead it keeps, per region, a set of tick queues indexed by interval. A check on a sixty-second interval lives in the sixty-second bucket for its region; on each wall-clock tick aligned to sixty seconds, the scheduler pops every check in that bucket and dispatches it to that region’s probes. This turns scheduling from N independent timers into a small number of buckets swept on a shared clock, which keeps wakeups coarse and predictable even with hundreds of thousands of checks. The practical consequence, matching how real systems run, is that a check with a ten-second frequency runs roughly every ten seconds and a check with a one-hour frequency runs roughly every hour, each from every probe assigned to it.
Probe assignment by consistent hashing and region affinity
When a check names a region, the scheduler still has to decide which specific probes inside that region run it. A naive round-robin would mean that every probe churns whenever the region’s probe set changes. The production answer is consistent hashing over the check id, mapped onto the region’s probe set. A check hashes to a position on a ring, and the probes whose tokens own the neighboring arc run it. When a probe joins or leaves the region, only the checks whose hash fell on that probe’s arc move; the rest stay put. Layered on top is region affinity: a check is anchored to a region first, and consistent hashing decides ownership within it, so an observation meant to come from EMEA keeps coming from EMEA even as the global probe fleet grows.
Rolling aggregation: error-budget burn rate
The aggregation engine receives a stream of success booleans, one per probe run, and must answer one question fast: is the service burning through its error budget faster than its SLO allows. It does this with a rolling window of fixed-duration buckets kept in memory. Each bucket counts successes and failures for a check. The window slides by expiring whole buckets as wall-clock passes, so querying the success rate over the last five minutes is a cheap sum over a handful of counters rather than a scan of raw results.
A burn rate of one means the service is failing exactly as fast as its SLO permits. A burn rate of fourteen means it is failing fourteen times too fast, which is the signal that pages an operator before the long-window SLO is actually breached. The windowed counter that backs this is examined next, under system hardening, because its memory behavior is what keeps the aggregation engine from choking.
Edge cases and chaos engineering
A monitor that works when the network is healthy is not done. Four failure modes separate a toy synthetic monitor from a production one.
Scheduler-to-probe partition: local scheduling fallback
A region’s probes hold a long-lived stream to their scheduler. When that stream breaks, because a transit provider is flapping or a regional firewall misconfigures, the scheduler can no longer dispatch. The wrong response is to stop probing, because the partition is precisely the event the monitor exists to observe. The right response is probe autonomy: each probe retains its last-received check definitions and keeps running them on their intervals against the target, recording results locally. The probe degrades from a remote-controlled checker into a self-scheduled one, and it reconciles back to the scheduler’s authoritative schedule when the stream returns.
A probe can only ever run checks it already knows about. It cannot invent new checks, because the scheduler is the sole authority over what the fleet observes. Autonomy is a keep-the-lights-on mode for known work, never a license to expand the observation set, which is why the probe discards its local guesswork the moment the scheduler’s schedule returns.
Probe-to-ingestion partition: buffer and forward
A harder split isolates the probes from the ingestion path while leaving the scheduler reachable. The probes keep receiving definitions and keep probing, but their results have nowhere to go. The design meets this with a local buffer on each probe: results accumulate in a bounded, disk-backed queue, and a forwarder drains the queue toward ingestion when the path heals. The queue is bounded on purpose. An unbounded buffer would let a long partition fill a probe’s disk, turning a network failure into a probe crash, which would then hide the recovery the monitor was supposed to report. When the buffer is full the probe drops the oldest results and records the drop count, so the aggregation engine knows the window has a gap rather than silently assuming no runs occurred.
Multi-tenant probe fairness
A probe is shared infrastructure. Two tenants on the same probe compete for its CPU and its network egress, and the loser’s checks run late, which looks identical to the target being slow. The defense is a per-tenant fair-share scheduler inside each probe: every interval, each tenant is granted a slice of the probe’s execution budget proportional to its allocation, and checks run within their tenant’s slice. A tenant that schedules thousands of checks cannot starve a tenant that schedules five, because its slice is the same size regardless of how much work it stuffed into it. The dispatch contract’s tenant id is what makes this enforcement possible: without it, the probe could not tell whose work to throttle.
Clock skew across regions
A result’s timestamp is the basis for which window it lands in. Probes in different regions have clocks that drift, and a result stamped a few hundred milliseconds off can land in the wrong bucket, which smears a failure spike across two windows and dulls the alert. The design treats the probe’s timestamp as an estimate, not a truth: the ingestion path restamps each result with its own receive time as a tie-breaker, and the aggregation engine uses the receive time, not the probe’s clock, to decide bucket membership. The nanosecond probe timestamp is retained for diagnosis and latency computation, where a constant skew across a check cancels out, but it is never the sole key for windowing.
System hardening and programming realities
The aggregation path is a system where the language runtime’s behavior is load-bearing. A garbage-collection pause of a few hundred milliseconds, invisible in a batch service, here means a stalled results stream and a delayed burn-rate update.
Garbage collection from high-frequency result parsing
The dominant allocation pressure on the results path is the deserialization of incoming OTLP batches: each push allocates result objects by the thousand. Left uncontrolled, this fills the young generation rapidly and triggers frequent collections, each of which stops the world long enough to cause backpressure spikes on the gRPC stream. The standard discipline is to pool buffers and reuse decoders, so a sustained push rate does not produce a sustained allocation rate. The windowed counters described next are pre-sized and fixed-bucket, so once the windows are warm the hot path allocates almost nothing.
The point of this structure is that its memory footprint is a compile-time constant: a counter for a check uses the same number of bytes whether it has recorded ten results or ten million. That is what keeps the aggregation engine’s resident memory proportional to the number of checks times the number of buckets, rather than to the raw result rate.
Aggregation window memory bounds
Because each windowed counter is fixed-size, the engine’s worst-case memory is predictable: it is the number of checks times the number of probes per check times the bucket count times the counter size. An unbounded result log, by contrast, would let a single noisy target inflate memory without limit. The defense is the same fixed-bucket discipline at every level: per-check, per-probe, and per-tenant windows all use the ring pattern, and a tenant whose check count explodes consumes more counters but never unbounded bytes per counter. When the engine approaches its memory ceiling it sheds the oldest per-tenant windows first, preserving global accuracy at the cost of per-tenant granularity, and records the shed so an operator knows the window degraded.
Backpressure on the results path
The results path is the one place a slow consumer can choke the whole system. If the aggregation engine falls behind, the ingestion path must slow the probes down rather than buffer the overflow in memory. It inherits this from the transport: gRPC over HTTP/2 advertises a flow-control window per stream and per connection, and a slow ingestion path that stops reading lets its receive window drain, which forces the probe’s HTTP/2 stack to pause its writes with no application-level code. The ack contract’s retry hint is the application-level reinforcement of the same idea: when the engine is saturated, ingestion tells probes how long to back off, so a recovering fleet does not stampede it all at once. The diameter of the flow-control window must cover the bandwidth-delay product of the link, or the stream stalls even when both ends are idle.
Architectural evolution: 10x, 100x, 1000x
A topology that is elegant at ten thousand checks becomes a bottleneck at ten million. The evolution is forced by specific saturation points.
At 10x: a single scheduler
The first saturation is rarely the probes; it is the single scheduler. One scheduler holding the global schedule, dispatching to every region, becomes both a CPU bottleneck and a single point of failure. The 10x response is to make the scheduler stateless-fronted: the schedule lives in a shared, replicated store, and any number of stateless scheduler instances read it and dispatch for their assigned regions. The aggregation engine, still single, keeps up because its per-result work is two increments. The read path, dashboards querying burn rates, gets a simple cache in front of it to absorb the amplified query load that 10x traffic brings.
At 100x: regional schedulers and sharded aggregation
At 100x, a single aggregation engine is the limit. The result river is too wide for one process to count. The response splits scheduling and aggregation along the same seam: regional schedulers own their region’s dispatch, and the aggregation engine is sharded by check id, so each shard owns a slice of the check space and its windows. A result is routed to its shard by the hash of its check id, the same hash the scheduler used to assign the check to probes, which keeps related work colocated. Multi-tenant fairness moves from per-probe slices into per-tenant quotas per shard, so a noisy tenant’s blast radius is bounded to the shards it lands on. Gossip propagates schedule changes between regional schedulers, so a check created in one region is eventually known everywhere.
At 1000x: edge-local autonomy and the cell
At 1000x, a single globally-shared schedule store and a single global shard map are no longer tenable. The architecture shifts to cells: independent, self-contained deployments of the scheduler, probes, ingestion, and aggregation stack, each owning a slice of tenants and a slice of regions. A regional edge routes a check definition to the cell that owns its tenant, so no cell holds global state. The scheduler-to-probe stream, which was already a long-lived regional connection, becomes a cell-local one. Synchronous coordination, the global schedule store and the global shard map, is replaced by an asynchronous check-definition fabric that propagates changes eventually rather than immediately. Probe autonomy, which was a fallback mode at smaller scale, becomes the default at the edge: each cell’s probes hold their definitions locally and keep probing through any partition, because there is no global authority to lose contact with.
A check that enters cell one cannot block a check in cell two, because they share no scheduler, no aggregation shard, and no synchronous path. The property that survives hyperscale is the same one that survived a partition: a failure in one place cannot take down the observation of another.
How Grafana does it
This design is not abstract; it is the shape Grafana Synthetic Monitoring takes. Grafana Synthetic Monitoring is a black box monitoring solution, built on top of the Prometheus blackbox exporter, that tests applications by running checks continuously against remote targets from probe locations around the world. Each execution of a check simulates a single user performing a single iteration from one location, which is exactly the lightweight, repeatable unit of observation this concept is built around.
Probes are the data plane. They are the agents responsible for emulating user interactions and collecting data from targets across global locations, and they come in two kinds: public probes, instances of the open source Synthetic Monitoring Agent operated by Grafana Labs at distributed locations, and private probes, instances of the same agent that you run yourself and which write only to your account. Private probes authenticate with a per-probe token and communicate with a regional API server over gRPC on port 443, which is the concrete instance of the scheduler-to-probe boundary described here. The execution model matches the design’s core rule precisely: when you create a check you select which probes run it, and each selected probe executes the check independently at every scheduled interval. Probes do not rotate, so a check with five probes on a one-minute frequency produces five executions per minute, one from each location, arriving concurrently at the target.
Checks are the unit the scheduler dispatches. Each check carries a job name and a target that together uniquely identify it, a set of probe locations, a frequency between ten and thirty-six hundred seconds, and a timeout between one and sixty seconds. The success signal is the blackbox exporter’s model in production form: after the configured interaction, the probe emits a probe success metric (and a probe duration seconds metric), which is the single decisive boolean the aggregation layer reduces everything from. Results are saved as Prometheus metrics and Loki logs, which feed the same alerting that the rest of Grafana Cloud uses, so a synthetic check’s failure pages an operator through the identical Grafana Alerting machinery that a threshold on an internal metric would. That is the proof of the design’s central claim: the control plane (scheduling) is deliberately decoupled from the data plane (probing), so that the act of observing a user’s experience survives the same network failures it exists to detect.
Mastery Questions
Question
Why does synthetic monitoring separate the scheduler (control plane) from the probes (data plane), instead of one service doing both?
Answer
Because the two must fail independently. The scheduler decides what to test and from where; the probe performs the test. If a probe also owned the schedule, losing contact with peers would leave it guessing at a schedule it could not reconcile. If the scheduler also probed, a region going dark would lose both the schedule and the only observer that could report the darkness. Splitting them means a probe cut off from the scheduler keeps probing its last-known checks, so the observation outlives the failure it observes.
Sources & evidence8 claims · 6 cited
All eight claims are backed by fetched primary sources (Grafana Synthetic Monitoring overview, intro, checks, private probes, alerts docs; Prometheus blackbox_exporter). The OTLP results-push contract is inferred from the documented gRPC probe channel rather than over-claimed as a spec; cell-based and consistent-hash-probe-assignment evolution are frontier reasoning with no claim.
- Synthetic monitoring tests applications by running checks continuously against remote targets from geographically distributed probe locations, because a service reporting its own health cannot see failures that occur between the user and the service (DNS, CDN, certificates, transit, regional filtering).verified
- Grafana Synthetic Monitoring is a black box monitoring solution built on top of the Prometheus blackbox exporter, which allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP, and gRPC.verified
- A check is uniquely identified by its job name and target, is assigned probe locations, has a frequency between 10 and 3600 seconds and a timeout between 1 and 60 seconds, and its success is expressed through the probe_success metric.verified
- The control plane (the scheduler that holds check definitions and the schedule) is decoupled from the data plane (the probes that perform the checks), so a probe that loses contact with the scheduler can continue running its last-known checks and keep observing through a partition.verified
- The blackbox exporter implements the multi-target exporter pattern in which the target is passed as a parameter and the probe timeout is derived from the scrape timeout, slightly reduced to allow for network delays.verified
- Each probe selected for a check executes that check independently at every scheduled interval; probes do not rotate or take turns, so a check with five probes on a one-minute frequency produces five executions per minute, one from each location, arriving concurrently at the target.verified
- Private probes are instances of the open source Synthetic Monitoring Agent that authenticate with a per-probe token and communicate with a regional API server over gRPC on port 443, with each region having a dedicated API server URL.verified
- Check results are saved as Prometheus metrics (including probe_duration_seconds and probe_success) and Loki logs, which feed the same Grafana Alerting machinery used by the rest of Grafana Cloud.verified
Cited sources
- Synthetic Monitoring (overview) · Grafana Labs
- Synthetic Monitoring Introduction · Grafana Labs
- Prometheus blackbox_exporter · Prometheus / CNCF
- Synthetic Monitoring Check types · Grafana Labs
- Set up private probes · Grafana Labs
- Synthetic Monitoring alerting · Grafana Labs