On this path · Control Plane 1. Alerting and Rule Evaluation Engine
Alerting and Rule Evaluation Engine
Continuous rule evaluation over a consistent hash ring with grouped evaluation loops and distributed deduplication.
Learning outcomes
An observability backend can store years of metrics perfectly and still fail its users, because what they actually want is to be told the moment something breaks. A dashboard shows a curve; an alert wakes a human. Turning a curve into a timely, correct wake-up call is a harder problem than storing the curve, because it demands continuous work that never stops, that must deduplicate itself across replicas, and that must survive the failure of the very process doing the checking. That problem forces alerting into its own continuously running evaluation service.
After studying this page, you can:
- Name the ruler, the query-frontend, and the Alertmanager, and trace both the rule-evaluation read lifecycle and the alert-delivery write lifecycle through them.
- Specify the proto3 contract for an EvaluateRules RPC and a PostAlerts RPC to Alertmanager.
- Describe how a consistent hash ring assigns rule groups to ruler replicas and why grouped evaluation loops batch rules that share a query.
- Explain the pending, firing, and resolved alert states and the FOR-duration timer that gates them.
- Diagnose ruler crashes mid-evaluation, split-brain duplicate firing, evaluation storms, and multi-tenant rule explosions, naming their mitigations.
- Reason about garbage collection, query-result caching across a group, and backpressure to Alertmanager as the real programming constraints.
- Map the 10x, 100x, and 1000x evolution onto Grafana Mimir’s ruler and Alertmanager.
Before we dive in
Picture a single Prometheus instance asked to keep watch over a fleet of a thousand services. An operator writes a few hundred alerting rules, each a PromQL expression like rate(http_requests_total{status="5xx"}[5m]) > 0.1, and expects a page the moment error rate climbs. To deliver that promise, the instance must re-evaluate every rule against fresh data, on a fixed interval, forever. It is not enough to run a query once when asked; the condition must be checked continuously, because the value that crosses the threshold does so between human glances.
The single-node evaluator dies there for a reason that has nothing to do with the query being slow. The moment it must run ten thousand rules across ten thousand series every fifteen seconds, the work no longer fits one process: one expensive rule starves the next, a single rule whose expression resolves a million series consumes the CPU for the whole interval, and a process restart drops every pending and firing alert on the floor. Scaling the box up only postpones the wall, because rule count and series count both grow with the fleet. Worse, a restart is a blackout: every FOR-duration timer resets, every pending alert forgets how long it has been nearly firing, and the human gets paged twice, or not at all.
The fix is not a faster evaluator. It is to stop doing the whole job in one process. Split the rules into groups, assign each group to a different ruler replica via a consistent hash ring, let each replica run its own tight evaluation loop, and hand the firing alerts to a separate Alertmanager whose only job is to deduplicate, group, and route them. That reframe, from one forever-loop to many sharded loops coordinated by a ring and relieved of delivery by a dedicated notifier, is what an alerting and rule evaluation engine is. Every component exists to make one part of that continuous work sharded, stateful where it must be, and survivable.
Microservices topology and component interaction
The alerting plane is a set of ruler replicas in front of a query path and an Alertmanager. The rulers do the continuous checking; the query path supplies the samples they check against; the Alertmanager does the delivery. Nothing on the ruler side delivers a notification to a human; nothing on the Alertmanager side evaluates a rule. That decoupling is what lets the two scale and fail on their own schedules.
Three roles matter, each earned by the problem it solves.
The ruler is the continuous evaluator. Each ruler replica joins a hash ring, claims ownership of a slice of rule groups, and runs an evaluation loop that re-checks its groups on their configured interval. It speaks gRPC to the query path to evaluate each rule’s expression, and it speaks HTTP to the Alertmanager to push any alerts that have crossed into firing. The ruler is the only component that holds the pending and firing state of an alert, which is why its placement on the ring and its survival across restarts both matter.
The query-frontend is the path the ruler uses to actually compute a rule’s expression. Evaluating rate(...) means fetching samples and running the PromQL engine over them, exactly as a dashboard does, so the ruler delegates that work to the same query-frontend and query path that serves dashboards. This delegation is what lets a rule benefit from query sharding and caching, and what keeps the ruler focused on scheduling and state rather than on sample fetching. In some deployments the ruler runs an in-process querier instead and talks directly to ingesters and store-gateways; the contract is the same, only the hop count differs.
The Alertmanager is the delivery layer. It receives firing alerts from every ruler, deduplicates them (so the same alert from three rulers becomes one notification), groups them (so a storm of related alerts becomes one page), and routes them to the right receiver. The Alertmanager is deliberately separate from the ruler because delivery has its own concerns: silencing, inhibition, rate limiting, and templating, none of which belong in the tight evaluation loop.
The ruler runs two distinct lifecycles. The read lifecycle evaluates a rule against fresh samples (ruler to query-frontend to querier to ingester and store-gateway). The write lifecycle pushes a firing alert outward (ruler to Alertmanager). They share no state except the alert’s label set, which is the deduplication key between them.
The wire between the rulers and the query path is gRPC carrying Protocol Buffer messages, because evaluation is a tight, typed, low-latency internal call. The wire between the rulers and the Alertmanager is HTTP carrying a JSON array of alerts, because delivery crosses an API boundary that the Alertmanager specifies independently of any one vendor. That protocol split is not accidental: the internal call wants performance and schema evolution, while the external call wants a stable, documented contract.
Rules within a single group are evaluated together in one batched pass. A rule group is the unit a user writes, and rules that share a group share a query context: their expressions often read the same underlying series, so evaluating them in sequence lets the ruler reuse a cached series result across rules in the group instead of re-fetching it for each rule. Grouped evaluation is therefore both a scheduling unit (one interval per group) and a performance unit (one shared fetch per group).
The evaluation and alert lifecycles step by step
A single rule moves through two lifecycles in a fixed order: evaluate to decide state, then push if firing.
The lifecycles earn each step. Tick is the group’s interval firing, which is the only thing that wakes the loop, so the ruler does no busy waiting between checks. Load pulls the group’s rules from the rule store once per tick. Evaluate delegates the expression to the query-frontend, which returns the vector of series currently satisfying the rule. Update is the moment state changes: a series first seen active enters pending, a series active long enough enters firing, a series no longer active resolves. Post pushes only the firing alerts to the Alertmanager over HTTP, carrying their label sets as deduplication keys. Schedule advances the next tick by the group interval, keeping the loop cadence stable even when one evaluation ran long.
Concrete API schemas
The evaluation contract must carry everything the ruler needs to check a group and everything the Alertmanager needs to deduplicate a firing alert: a tenant for isolation, the rule group and its interval, the expressions and their FOR durations, and for delivery the label set that identifies each alert uniquely.
The split between an evaluation RPC over gRPC and an alert-push RPC that maps to an HTTP POST is deliberate. Evaluation is an internal, typed, latency-sensitive call that benefits from schema evolution; delivery crosses the Alertmanager API boundary, where a stable JSON contract matters more than wire efficiency.
Low-level data structures and execution
The evaluation loop scheduler
Each ruler replica runs one goroutine per rule group it owns. The group is the scheduling unit: it has one interval, one next-tick time, and one rule list that evaluates together. The loop is not a busy spin; it sleeps until the next tick, evaluates, then reschedules.
The scheduler earns its details. The fingerprint map is what makes the loop stateful: without it, every tick would forget that a series has been pending for nine of its ten required minutes, and the alert would never fire. The cached series fetch across rules in one group is what makes evaluation cheap: a group of ten rules reading the same metric fetches it once.
Hash-ring rule ownership
A ruler replica does not evaluate every group. It evaluates only the groups the hash ring assigns to it, which is what lets the rule load fan out across replicas.
The ring is the consistent hash ring the rest of the platform already uses. The group id is hashed to a 32-bit token, and the ruler registered with the smallest token greater than it becomes the owner, wrapping around at the top of the space. When a ruler joins or leaves, only a bounded slice of groups moves, on average the token count divided by the instance count, so a rolling restart does not reshuffle the whole fleet. The rulers heartbeat their liveness into the ring stored in a shared key-value store, so an unhealthy replica’s groups are transparently picked up by a healthy one.
Alert state and the FOR-duration timer
The heart of the engine is the per-series state machine. An alert is not a boolean; it is a timeline that spends time in pending before it earns firing, and that must be actively resolved when the condition clears.
A series first seen satisfying a rule enters pending and is stamped with an active-since time. On every subsequent tick where it is still active, the ruler compares the elapsed time against the rule’s FOR duration. Only when the series has been continuously active for the entire duration does it graduate to firing, at which point the ruler posts it to the Alertmanager. A series that drops out of the active set while still pending simply disappears: it never fired, so there is nothing to resolve. A series that was firing and then leaves the active set is resolved, and the ruler sends a final alert with an end timestamp so the Alertmanager closes the notification cleanly.
The FOR duration is the mechanism that turns a flapping metric into a trustworthy signal. Without it, a single noisy evaluation would page a human; with it, the alert must prove the condition is sustained before it earns the right to interrupt someone.
The inequality shows why the interval matters as much as the duration: a FOR duration of ten minutes checked every fifteen seconds fires within ten minutes plus at most fifteen seconds, while the same duration checked every five minutes could fire up to five minutes late. Tighter intervals give more accurate timing at the cost of more evaluation work.
Edge cases and chaos engineering
Ruler crash mid-evaluation
A ruler that crashes between ticks drops every pending timer and every firing alert it owned. Because the ring tracks liveness, the crashed replica’s groups are quickly reassigned to a healthy replica, which resumes evaluating them. The subtle part is the lost state. The new owner has no record of how long a series had been pending, so it restarts the FOR-duration clock from zero: an alert that was one tick away from firing now waits the full duration again.
The mitigation is to make the ruler as stateless as the design allows. The rule definitions themselves live in a shared rule store, not in the ruler’s memory, so any replica can pick up any group. The alert state is the only thing that does not trivially survive, which is why firing alerts are re-sent to the Alertmanager on every tick: the Alertmanager, not the ruler, is the durable record of what is currently firing. A resolved notification may be delayed, but a firing alert is never silently lost, because the new owner re-discovers the condition on its very next evaluation and re-posts it.
Split-brain and duplicate firing
When multiple ruler replicas briefly own the same group, which can happen during a ring transition or a network partition, both evaluate it and both post the same firing alerts to the Alertmanager. The ruler plane does not try to prevent this; it accepts that duplication is possible and relies on the Alertmanager to collapse it.
Alertmanager deduplicates alerts that share an identical label set, so two rulers posting the same alert normally collapse to one notification. Under a genuine network partition, the Alertmanager cluster itself can split, and its design chooses to fail open: it prefers to send a duplicate notification than to risk missing a critical alert. The tradeoff is deliberate, because in alerting a false silence is far worse than a false repeat.
Evaluation storms and Alertmanager flooding
A single failing dependency can make hundreds of rules fire at once. A database becomes unreachable, and every service that depends on it trips its error-rate alert in the same evaluation window. Every ruler posts its firing alerts, and the Alertmanager is suddenly the bottleneck.
The defenses are layered. The Alertmanager groups alerts that share labels (typically alertname and cluster), so a hundred related firings become one notification with a list inside it, rather than a hundred separate pages. A per-alertname limit caps how many active alerts of one name the Alertmanager will track, dropping new ones once the limit is reached while still processing heartbeats from alerts already known, so one runaway rule cannot exhaust its memory. On the ruler side, posting is asynchronous and bounded: a ruler whose Alertmanager is slow applies backpressure rather than queuing an unbounded number of firing alerts in memory.
Multi-tenant isolation
A shared cluster serves many tenants, and one tenant with a runaway rule set can starve the rest. A tenant who writes a rule whose expression matches a million series, or who schedules a thousand groups at a one-second interval, can monopolize the ruler replicas that own its groups.
The isolation comes from the ring combined with per-tenant limits. Shuffle-sharding confines each tenant’s groups to a subset of rulers, so a noisy tenant’s blast radius stays inside its own shard and cannot crowd out the groups of quieter tenants on the same replica. Per-tenant limits cap the number of rules, the evaluation frequency, and the cost of any single rule’s expression, so a pathological rule is rejected with an error rather than allowed to consume the replica.
System hardening and programming realities
The realities of running a forever-loop at scale are less glamorous than the topology but determine whether the engine survives a busy week.
Garbage collection from expression parsing. Every evaluation that delegates to the query path parses a PromQL expression, builds an AST, and allocates short-lived objects for the result vector. A ruler evaluating a thousand rules every fifteen seconds produces a steady stream of garbage that pauses the loop. The discipline is to parse each rule once and cache the parsed plan across ticks, and to ensure the result vector is consumed and released within the tick rather than held across the sleep. The same reasoning bounds the alert state map: fingerprints for resolved series must be expired, or the map grows monotonically with every series the fleet ever produced.
Caching query results across rules in a group. Rules grouped together often read the same underlying series, so the ruler evaluates them in an order that lets a shared fetch be reused. The first rule in a group that reads http_requests_total fetches and caches the series; subsequent rules reading the same metric serve from the cache within the same evaluation pass. The cache is scoped to the tick, not persisted, so it never goes stale across intervals.
Backpressure to Alertmanager. The HTTP post to the Alertmanager is the one place the ruler hands work outward, and an unbounded outbound queue is an out-of-memory condition waiting to happen. A bounded queue with a retry limit means that when the Alertmanager is saturated, the ruler drops or delays the post and logs it, rather than buffering every firing alert until the replica dies. Because firing alerts are re-sent every tick, a dropped post is recovered on the next interval, so backpressure costs timeliness, not correctness.
Sharding the ring evenly. A hash ring is only as balanced as its tokens. With too few tokens per instance, one ruler can own a disproportionate slice of groups and become the hot replica. The ring uses many tokens per instance and a spread-minimizing strategy so that group ownership is roughly even, and the workload distributes across the replicas rather than piling onto one.
Architectural evolution: 10x, 100x, 1000x
At 10x: a single ruler
At modest scale the alerting plane is one ruler process evaluating a local rule file on a fixed interval and posting firing alerts to a single Alertmanager. The rules are few enough that one loop handles them, and the Alertmanager absorbs the occasional burst. The failure mode here is the restart: the single ruler holds every pending and firing state in memory, so a crash resets every FOR-duration timer and briefly loses the record of what was firing. The lesson learned is that alert state must live somewhere more durable than one process, and that delivery must be decoupled from evaluation.
At 100x: sharded rulers on a ring plus dedup
Growth forces the split. The rulers become a pool that joins a hash ring, each owning a slice of rule groups stored in a shared backend, so adding replicas spreads the evaluation load. Multiple rulers can now post the same alert, so the Alertmanager’s label-set deduplication becomes load-bearing rather than incidental. The rule store moves to object storage or a database so any replica can pick up any group after a failure. The new failure mode is split-brain duplication and state loss on reassignment, which is what drives the fail-open Alertmanager design and the re-send-every-tick discipline.
At 1000x: cells, per-tenant rule stores, and an async alert fabric
At the largest scale a single cluster hits limits on the ring, the rule store, and the blast radius of a failure. The architecture splits into cells, each a self-contained alerting plane with its own rulers, query path, and Alertmanagers. Rule stores become per-tenant so one tenant’s rule explosion cannot degrade the shared store. An asynchronous alert fabric appears, where high-volume alerts are routed through a buffering layer that rate-limits and de-duplicates before they reach the Alertmanager, smoothing the storms that a single cell cannot absorb. The throughline is the same: shard the evaluation by ring, deduplicate by label set, and keep the delivery layer separate from the tight evaluation loop.
How Grafana Mimir does it
Grafana Mimir is the canonical embodiment of this design, and its alerting plane maps almost one-to-one onto the topology above.
The Mimir ruler is the continuous evaluator. It evaluates the PromQL expressions defined in recording and alerting rules for each tenant, grouped into namespaces, and at regular intervals. Rule groups are the unit of sharding: ruler replicas form their own hash ring, stored in the key-value store, to divide the work of evaluating rule groups across the pool, so each replica owns only a slice of the total rule load. The ruler supports two evaluation modes. In internal mode it runs an in-process querier and connects directly to ingesters and store-gateways. In remote mode it delegates evaluation to the query-frontend over gRPC, which lets it benefit from query acceleration techniques such as query sharding.
The alert lifecycle inside the Mimir ruler follows the state machine exactly. When an alerting rule’s expression returns any series, the alert becomes active. If the rule defines a FOR duration, it enters the pending state and only transitions to firing after remaining active for the entire duration, at which point the ruler notifies the configured Alertmanagers. The ruler is pointed at the Alertmanager API with the -ruler.alertmanager-url flag, and the wire is HTTP to the Alertmanager endpoint whose prefix defaults to /alertmanager.
The Mimir Alertmanager is the delivery layer. It accepts alert notifications from the Mimir ruler, then deduplicates, groups, and routes them to notification channels. It adds multi-tenancy and horizontal scalability to the Prometheus Alertmanager: replicas join their own hash ring and shard and replicate alerts by tenant, so any replica can respond to any tenant’s request and the load spreads across the pool. The Alertmanager holds alert state on local disk and periodically snapshots it to a storage backend, so on restart it can recover a tenant’s state from peer replicas or fall back to the snapshot.
The deduplication between ruler replicas and across Alertmanager replicas is the seam that holds the plane together. Because every ruler that owns a group re-posts its firing alerts on every evaluation, the same alert can arrive from more than one source; the Alertmanager collapses any alerts that share an identical label set into a single notification. And because the Alertmanager cluster is designed to fail open during a network partition, it prefers to deliver a duplicate notification than to risk a missed alert, trading a small chance of repetition for a strong guarantee of delivery.
Mastery Questions
Question
A single-node ruler restarts and every pending alert forgets how long it has been nearly firing. Why does the sharded design still avoid losing firing alerts, even though it cannot preserve pending timers across a crash?
Answer
Firing alerts are re-posted to the Alertmanager on every evaluation tick, so the Alertmanager, not the ruler, is the durable record of what is firing. When a crashed ruler's groups move to a healthy replica on the hash ring, the new owner re-evaluates and re-posts within one interval. Pending timers do reset to zero on reassignment, so an alert about to fire may be delayed, but a genuinely firing condition is never silently lost because it is rediscovered on the next tick.
Sources & evidence9 claims · 7 cited
All nine claims are backed by fetched primary sources (Grafana Mimir ruler, Alertmanager, and hash-ring docs; Prometheus alerting-rules and Alertmanager docs). The 1000x cell-based async alert fabric is frontier design reasoning, not a documented Mimir feature, and carries no claim.
- Alertmanager receives alerts from clients such as the ruler, then deduplicates them by label set, groups alerts of similar nature into a single notification, and routes them to receivers; this is why delivery is a separate layer from evaluation.verified
- An Alertmanager cluster communicates over a gossip protocol and replicates silences and the notification log; under split-brain it fails open, preferring to send duplicate notifications over missing critical alerts, with at-least-once delivery.verified
- Clients are expected to re-send firing alerts to Alertmanager at regular intervals, and firing alerts resolve once their endsAt timestamp has elapsed; Alertmanager sets endsAt to the current time plus resolve_timeout when omitted.verified
- An alerting rule with a defined for duration enters the pending state when its expression first returns a series, and only transitions to firing after remaining active for the entire for duration, at which point the ruler notifies the configured Alertmanagers.verified
- A Mimir hash ring hashes the workload to a 32-bit token owned by the instance with the smallest greater token; when an instance joins or leaves, on average only the token count divided by the instance count of work moves.verified
- In remote mode the Mimir ruler delegates rule evaluation to the query-frontend over gRPC, which lets rules benefit from query acceleration techniques such as query sharding; in internal mode it runs an in-process querier against ingesters and store-gateways.verified
- The Mimir ruler shards the execution of rules by rule group: ruler replicas form their own hash ring stored in the KV store to divide the work of evaluating rule groups across the pool.verified
- The Grafana Mimir ruler evaluates the PromQL expressions defined in recording and alerting rules at regular intervals and passes the resulting samples to an in-process distributor for ingestion.verified
- The ruler is configured with Alertmanager addresses via the -ruler.alertmanager-url flag, pointed at the Alertmanager API whose prefix defaults to /alertmanager, and posts firing alerts to it over HTTP.verified
Cited sources
- Prometheus Alertmanager (grouping, inhibition, silencing, routing) · Prometheus Authors (CNCF)
- Grafana Mimir Alertmanager (multi-tenant, sharded, accepts ruler notifications) · Grafana Labs
- Alertmanager High Availability (gossip, fail-open, dedup via notification log) · Prometheus Authors (CNCF)
- Alertmanager Alerts API v2 (POST alerts, dedup by labels, resolve_timeout) · Prometheus Authors (CNCF)
- Prometheus Alerting Rules (for, pending, firing, keep_firing_for) · Prometheus Authors (CNCF)
- Grafana Mimir Ruler (rule evaluation, sharding, Alertmanager integration) · Grafana Labs
- Grafana Mimir Hash Rings (consistent hashing, sharding, replication) · Grafana Labs