On this path · Platform 1. Multi-Tenant Authz and RBAC Service
Multi-Tenant Authz and RBAC Service
Sub-millisecond authorization with hierarchical permission caching and gossip-spread eventual consistency.
Learning outcomes
Authorization is the check that runs on every single request, far more often than authentication, and at a volume that makes a database lookup on each one impossible. This page teaches the service that answers that check in under a millisecond, for thousands of tenants, while policy keeps changing underneath it.
After studying this page, you can:
- Name the services on the authorization path and explain why a policy decision point is fronted by a cache and enforced by embedded policy enforcement points.
- Specify the Authorize RPC and the signed token it consumes.
- Map the hierarchical permission cache, the role-to-permission resolution engine, and the bitmap and Bloom-filter fast paths.
- Describe how a signed token is verified with no database lookup, and how a revocation spreads through gossip with version vectors.
- Diagnose revocation lag, invalidation races, gossip partition, token replay, and cross-tenant escalation.
- Reason about the garbage-free hot path, precomputed bitmaps, constant-time checks, and caching TTL tradeoffs.
- Trace the 10x, 100x, and 1000x evolution toward regional decision points and cell-local policy stores.
- Ground the design in Grafana RBAC, its basic, fixed, and custom roles, and its folder hierarchy.
Before we dive in
Picture an observability platform serving a million queries a second across thousands of organizations. Every one of those queries must first answer a question: is this subject allowed to read this dashboard, in this folder, for this tenant? The naive answer is to ask the database. On every request, look up the user, walk their role bindings, resolve the permissions, and compare against the resource. At one million queries a second, that is one million database round trips just to decide whether a request may proceed, before the request itself does any work. No database survives that. The check would cost more than the work it guards.
The problem that forced a dedicated authorization service into existence is that the check is on the hottest path in the system, it must be correct, and the policy it encodes never stops changing. An administrator revokes a role mid-incident; a team is granted access to a new folder; a token expires. The decision cannot be cached forever, because policy is mutable, yet it cannot hit the database, because the volume is too high. Those two facts, mutability and volume, are in direct tension, and resolving that tension is the entire job of this service.
The way out is to split the decision into two layers. A policy decision point, the PDP, holds the full policy and computes authoritative answers. In front of it sits a fast cache, and beside every caller sits a policy enforcement point, the PEP, that asks the question and enforces the answer. The subject carries a signed token, a compact, self-describing credential whose validity can be checked with a signature and an expiry alone, with no database lookup at all. The cache absorbs the repeated decisions for the same subject, resource, and action, so the million queries a second collapse to a small number of distinct cache keys. And because policy changes are rare relative to decisions, the cache can be invalidated lazily, by spreading revocation events over gossip, accepting that a revoked permission is honored briefly until every PEP hears the news.
That last concession is the heart of the design. Authorization at this scale is not synchronous. It is eventually consistent. The service trades a small, bounded window of over-permission for the ability to answer in microseconds, and every mechanism on the page, from the signed token to the version vectors on the gossip channel, exists to keep that window short and that over-permission safe.
Microservices topology and component interaction
The authorization path separates cleanly along the line where a slow, authoritative decision is copied into a fast, cached one. The policy decision point computes truth. The cache repeats truth. The policy enforcement points ask and obey. The policy store holds the source of policy. Nothing on the hot path reaches the store directly.
The key thing to read from this picture is that the policy store sits off the hot path: it feeds the PDP only when policy changes, while every request resolves inside the caller’s own cell through the cache or a single PDP hop.
PDP, cache, PEPs, and the policy store
The policy decision point is the single authority over what is allowed. It loads the full policy, the set of roles, their permission actions and scopes, and the role bindings that attach them to subjects, into memory, and it answers an Authorize request by resolving those bindings to a permission set and matching the requested action and resource. Because the policy is held in memory, a decision costs no disk access; the store is consulted only when policy changes, not on the hot path.
The decision cache sits in front of the PDP and is the reason the system survives the volume. A cache entry is keyed by the tuple of tenant, subject, resource, and action, and it stores the decision plus a version stamp drawn from the policy at decision time. The vast majority of decisions repeat, so a hot subject hitting the same dashboards resolves in the cache without ever touching the PDP. The policy enforcement points are not separate services. They are libraries embedded inside each observability microservice, so the call to ask for a decision never crosses the network twice. A PEP takes the token and the requested action, consults the shared cache, and either proceeds or denies, locally.
Sub-millisecond decisions and the wire token
The decision must return in well under a millisecond, because it runs ahead of every query. A network round trip to a remote decision service would alone consume the budget, which is why the PEP is embedded and the cache is local. The subject is identified not by a database-fetched identity but by a signed token carried on the wire, a compact, self-describing credential in the style of a JSON Web Token, or a capability token such as a macaroon or biscuit. The token binds a subject, a tenant, a set of roles, and an expiry, all under a signature the PEP can verify with a public key or shared secret it already holds. No lookup is needed to trust it.
How policy updates propagate
Policy is mutable, and a cached decision is only valid as long as the policy that produced it has not changed. Rather than invalidate eagerly, which would saturate the network on every edit, the system spreads policy changes lazily over gossip. When an administrator edits a role or revokes a binding, the PDP bumps a version and emits an invalidation event. That event propagates peer to peer through the PEP mesh, each of which drops the affected cache entries and refills them from the PDP on the next miss. Gossip is eventually consistent, so for a short window a revoked permission may still be honored by a PEP that has not yet heard. The whole architecture is engineered to make that window short and bounded, not to eliminate it.
Concrete API schemas
The decision contract carries the tenant, the subject and its roles, the resource, the action, and any request context that affects scope, and it returns an allow or deny together with a reason. The subject’s identity and roles travel in the signed token, but the Authorize RPC is what the PEP calls once the token is verified, so the schema is designed to be flat, cheap to marshal, and stable to extend.
The token itself is the credential that vouches for the subject id and roles in that request. In the JSON Web Token style it is three base64url parts, a header naming the algorithm, a payload of claims, and a signature over the first two, all separated by periods.
The split between the two schemas is deliberate. The token vouches for who the subject is and what they were granted at issue time; the Authorize RPC asks whether that grant still permits a specific action now. Together they let the PEP trust the caller from the signature alone and resolve the decision from the cache, with the database out of the loop on both counts.
Low-level data structures and execution
The hierarchical permission tree
Inside the PDP, policy is not a flat list. It is a tree, rooted at the tenant, branching into roles, then into the permission actions each role grants, and finally into the scopes, the resource selectors, where each action may be applied. A scope can be exact, a single folder by its id, or a wildcard, the star form that matches every resource of its kind. This nesting mirrors how operators actually think about access, per organization, per team role, per folder, which is also how a real system such as Grafana scopes a permission to a folder and lets that grant cascade to everything beneath it.
Resolving a request is a walk. Starting at the tenant, the engine selects the roles bound to the subject, descends to the action being requested, and asks whether any granted scope matches the resource. A scope on a parent folder implicitly covers its children, because the matching rule treats a resource path as covered by any ancestor scope, exactly the cascade a folder permission system relies on. The first matching allow wins; absent a match, the default is deny.
Role binding to permission resolution
A role binding is the edge that attaches a role to a subject inside a tenant. Resolving a decision means turning the subject’s bound roles into the concrete set of action and scope pairs the tree holds, then testing membership. Because the tree is stable for a given policy version, the resolved permission set for a subject can be computed once per version and reused, which is what makes the decision cheap enough to repeat a million times a second.
The match the engine performs is a scope match, testing whether a requested resource falls under a granted scope, with wildcards and cascade handled by prefix-style comparison on the resource path.
The covers relation, written as the triangle above, is where the hierarchy pays off. A wildcard scope covers everything beneath it, and a folder scope covers its subfolders, so a single grant at a high level answers many specific requests without enumerating them.
Fast role membership and bitmap checks
Walking the tree on every decision would still be too slow at the peak. Two data structures collapse the walk to a handful of bitwise operations. The first is a Bloom filter over each subject’s role set, consulted to reject, in constant time, the common case where a subject does not hold a role at all. A Bloom filter can yield a false positive, so a positive is confirmed by the second structure, a precomputed permission bitmap, a fixed-width integer whose bits correspond to permission actions. If a subject holds a role, the bitmap is the bitwise OR of the bitmasks of every role they hold, and testing an action is a single bit check. Resolving a decision becomes: filter to reject non-members, then bitmap to confirm the action, then scope-match only for the resources, which is far rarer than the action check.
Token verification with no database lookup
The token is verified entirely from data the PEP already holds. The PEP keeps the issuer’s public keys in memory, refreshed on a slow schedule, and verification is three local checks: recompute the signature over the encoded header and payload and compare, confirm the not-before and expiration against a local clock, and confirm the audience names this service. None of these steps touches the policy store. This is what makes the token cheap to trust at volume, and it is also what creates the revocation problem, because a validly signed, unexpired token remains trusted until something actively overrides it.
Eventual consistency of revocation
A revocation is the one event that must reach the PEPs despite the token being self-validating. When a binding is revoked, the PDP cannot rewrite the token, so instead it spreads the revocation as an event. Each event carries a version vector, a map from PDP node to a monotonically increasing counter, so a PEP can tell whether an event is newer than the policy generation its cached decisions reflect. The PEP compares incoming vectors by component-wise maximum, drops cache entries whose version is dominated, and accepts that until the event arrives, a stale entry may still allow. The trade is explicit: the token is trusted locally and instantly, while revocation is trusted globally and eventually.
Edge cases and chaos engineering
Revocation lag
The clearest failure mode is the window itself. A malicious insider’s role is revoked at noon, but a PEP in another cell does not receive the gossip event until noon plus two seconds. Inside that window, the revoked permission is still honored, because the token is still signed and unexpired and the cache entry still reflects the old policy. The defense is not to pretend the window is zero; it is to bound it. A short token expiry caps how long any over-permission can last, a revocation is fanned out urgently through the gossip mesh, and high-sensitivity actions can demand a synchronous PDP check that bypasses the cache entirely. The design tolerates brief over-permission for ordinary actions and pays the synchronous cost only where the risk justifies it.
Cache invalidation races
A cache entry can be filled and invalidated concurrently, and the race is subtle. A PEP miss triggers a PDP fetch, but before the response arrives a revocation event invalidates the key, and the now-stale response fills the entry back in, resurrecting a revoked permission. The defense is to stamp every cached decision with the policy version observed at fill time and to reject any fill whose stamp is older than the version the PEP last learned from gossip. A late fill then compares its version to the current vector and discards itself, so a revocation always wins the race against a fetch it predates.
Gossip partition and split-brain policy views
If the gossip mesh partitions, the two sides diverge in what they believe the policy is. One side has heard a revocation; the other has not. Both continue to serve decisions, so a subject may be denied on one side and allowed on the other, a split-brain policy view. The system does not attempt to force agreement during the partition. Instead it prefers the safer failure: when the mesh heals, the side with the newer version vector wins, revocations propagate and dominate, and the transient inconsistency collapses. For actions that cannot tolerate even brief inconsistency, the request must be routed to the cell that owns the policy shard for that tenant, accepting a latency cost to buy linearizability on the decisions that need it.
Token replay and forgery
A stolen token is as good as the original until it expires, which is why the jti claim and a short expiry matter. Replay defense keeps a time-bounded seen-set of consumed jti values for tokens that must be single-use, or a revocation list of jti values for tokens invalidated before their natural expiry. Forgery is stopped at the signature: a token whose claims have been tampered with fails the signature check, and a constant-time comparison prevents a timing oracle from leaking information about how far a forged signature differed from a valid one. The signing keys are rotated on a schedule, with old keys accepted only for a grace period shorter than the shortest token expiry.
Multi-tenant privilege escalation
The most dangerous bug is one that lets a subject in one tenant act in another. The defense is structural, not procedural. The tenant id is part of every cache key, part of every role string, and part of every scope, so a role granted in tenant acme cannot match a resource scoped to tenant beta because the prefixes never align. The Authorize RPC rejects any request whose token tenant does not equal the request tenant before the policy engine is even consulted, so escalation requires defeating the signature, not finding a clever scope. Per-tenant policy shards, where each tenant’s policy lives on a distinct set of PDP nodes, add a physical boundary on top of the logical one.
System hardening and programming realities
A garbage-free hot path
The Authorize decision is called more often than any other function in the platform, and in a garbage-collected language the allocator and the collector are the first things to break under that load. A decision that allocates a map for the context, a slice for the reasons, and a string for the resource path will, a million times a second, saturate the allocator and trigger stop-the-world pauses that blow the sub-millisecond budget. The remedy is a hot path engineered to allocate nothing. The request is parsed into a reused buffer, the roles are read as views over the token’s bytes rather than copied strings, and the decision is returned in a value type, not a heap pointer. Steady-state allocation on the decision path is zero, because the only thing worse than authorizing a million requests a second is collecting the garbage they produce.
Precomputed permission bitmaps
The bitmap is what makes per-action checks constant time. Each permission action is assigned a stable bit index, and each role carries a precomputed mask, the OR of its action bits, computed once when the policy is loaded. A subject’s effective mask is the OR of the masks of their bound roles, computed once per policy version and cached. Testing whether a subject may perform an action is then a single bitwise AND against a precomputed integer, a nanosecond operation with no tree walk. When policy changes, only the masks for the affected roles and the subjects holding them are recomputed, so a policy edit costs work proportional to its blast radius, not to the size of the tenant.
Constant-time signature checks
Signature verification compares a computed value against an expected one, and a naive comparison that short-circuits on the first differing byte leaks information about how close a forgery came. The fix is a constant-time comparison that always reads every byte and reveals nothing about where, if anywhere, the values matched. The same discipline applies to scope matching for sensitive resources, so a caller cannot probe the policy by timing how long a denial takes. Cryptographic correctness here is also performance correctness: a constant-time path has no data-dependent branches, which is friendlier to the branch predictor on a path that runs this hot.
Caching TTL tradeoffs
Every cached decision carries a TTL, and the choice of TTL is a direct trade between latency and revocation responsiveness. A long TTL means fewer PDP fetches and lower p99 latency, but it stretches the window in which a stale decision can survive. A short TTL shrinks the revocation window but pushes more load onto the PDP. The resolved compromise layers two TTLs: a soft TTL after which the entry is refreshed in the background without blocking the request, and a hard TTL after which the entry is served stale only if the PDP is unreachable, failing open for availability on read paths and failing closed for writes. The soft refresh hides latency from the caller, while the hard bound caps how stale any decision can get even under failure.
Architectural evolution: 10x, 100x, 1000x
At 10x: single PDP plus cache
At modest scale the whole policy fits in one PDP’s memory, and the first bottleneck is simply the PDP doing too much work per request. The 10x response is to add the decision cache in front of it and to start precomputing the permission bitmaps so a decision is a bitmap check rather than a tree walk. At this scale the PEPs are already embedded, but the cache is shared and centralized, and policy changes are pushed to the single PDP synchronously. The system is still one logical decision point; the fixes are internal to it.
At 100x: PEP-embedded decisions and gossip invalidation
At 100x, the centralized cache becomes the limit. Every PEP routing through one cache pays a network hop and creates a single contention point. The response is to push the decision cache into each PEP, so the common decision resolves locally with no network at all, and to replace synchronous policy push with gossip invalidation. Each PEP now holds its own slice of the cache, refreshed lazily from the PDP on a miss, and revocations spread peer to peer. This is the scale where eventual consistency becomes a first-class design constraint, where the version vectors on gossip events start mattering, and where the architecture accepts a bounded revocation window as the price of local decisions.
At 1000x: regional PDPs and cell-local policy stores
At 1000x, a single global PDP cannot serve a planet-spanning deployment, and a single policy store cannot be the source of truth for every tenant. The response is to shard by tenant into cells, each owning a slice of tenants with its own PDP, its own policy store, and its own PEP mesh, so a tenant’s decisions never leave its cell. Cross-region propagation becomes eventual: a policy edit made in one region spreads to regional PDPs asynchronously, and the version vectors reconcile across regions the same way they reconcile within a cell. The macro-shift is that there is no longer one decision point or one policy; there are many, coordinated by version-vector causality rather than by a synchronous lock, and the system trades global linearizability for regional autonomy and survival under partition.
Notice what the cell boundary removes: a tenant’s decisions no longer depend on a PDP an ocean away, so the sub-millisecond budget survives geographic distance, and a regional outage degrades only that region’s tenants rather than the whole platform.
How Grafana does it
This design is the shape of Grafana’s access control. In Grafana a user is associated with a role that includes permissions, and permissions determine the tasks a user can perform, which is the same subject, role, permission decomposition this service resolves at speed. Grafana distinguishes Grafana server administrator permissions, organization permissions, and dashboard and folder permissions, and organization permissions are global within an organization: an editor can see and update all dashboards unless specifically restricted. That nesting of server, organization, and folder scopes is exactly the tenant to folder resource hierarchy the permission tree encodes.
Grafana’s RBAC provides a standardized way of granting, changing, and revoking access so users can view and modify resources. It offers three kinds of roles. Basic roles are the standard roles available in Grafana’s open-source edition, and every user must have one assigned. Fixed roles are discrete Enterprise and Cloud roles that cannot be changed or deleted, assignable to users, teams, and service accounts for more granular control than basic roles. Custom roles are unique combinations of permission actions and scopes, created when the fixed set does not fit. This basic, fixed, custom layering is the same role binding to permission resolution the engine performs: a basic role resolves to its associated fixed roles, each fixed role resolves to a bundle of permission actions and scopes, and the decision is the match of a requested action against that resolved set.
The permission itself is an action plus a scope, the two-tuple at the heart of the policy. An action describes what task a user can perform, such as reading dashboards, and a scope describes where the action can be performed. Scopes use wildcards: a folders star scope matches any folder, a folders uid followed by an identifier matches one folder, and a permission granted to a folder cascades down to its subfolders. Folders are the primary way Grafana organizes and controls access, and a user’s effective access to a resource is determined by their folder permission level. A permission granted on a folder cascades to all subfolders and to all resources in those folders, and you cannot grant a user lower permissions on a subfolder than they have on a parent. Folders support three levels, View, Edit, and Admin, each including all capabilities of the levels below it, which is precisely the wildcard-and-cascade covers relation the permission tree evaluates.
Grafana Enterprise adds fine-grained access control on top. Data source permissions restrict which users, service accounts, and teams may query a given data source, and dashboard permissions specified on a dashboard or folder override the organization permissions for that entity. The RBAC scope permissions type delegate lets a user delegate only a subset of their own permissions by creating a new role, which is the same principle the service uses to keep escalation structural rather than procedural. Together these mechanisms are the concrete instance of the principle this page follows: split the decision into a fast cached layer and a slow authoritative one, carry identity in a signed token, and let the resource hierarchy do the work of making a few high-level grants answer millions of specific questions.
Mastery Questions
Question
Why can the authorization check not simply query the database on every request, and what two facts create the tension the authz service resolves?
Answer
At a million queries a second, a database round trip per check would cost more than the work it guards, and no database survives that volume. The two facts in tension are that policy is mutable (roles get revoked, tokens expire) yet the check runs on the hottest path and cannot afford a lookup. The service resolves this by splitting a slow authoritative policy decision point from a fast cached one and carrying identity in a self-verifying signed token, accepting eventual consistency for revocation.
Sources & evidence9 claims · 6 cited
All nine claims are backed by fetched primary sources (Grafana RBAC overview, roles and permissions, folder access control, actions and scopes, role definitions docs; RFC 7519 JWT). The PDP/PEP/cache topology, Bloom/bitmap hot path, version-vector gossip revocation, GC-free hot path, and 10x/100x/1000x evolution are taught from first principles and carry no claim.
- Each Grafana basic role is comprised of a set of associated fixed roles: Viewer holds only read roles, Editor includes Viewer roles plus creator and writer roles, and Admin includes Editor roles plus writer roles for data sources, teams, and permissions.verified
- Grafana Enterprise adds data source permissions restricting which users, service accounts, and teams may query a data source, and the RBAC delegate scope lets a user delegate only a subset of their own permissions by creating a new role.verified
- A permission granted on a Grafana folder cascades to all subfolders and all resources in those folders, folder permissions support View, Edit, and Admin levels each including the lower levels, and inheritance flows downward so a user cannot hold lower permissions on a subfolder than on its parent.verified
- The JWT exp claim identifies the expiration after which the token must not be accepted, the jti claim provides a unique identifier usable to prevent replay, and if any validation step fails the JWT must be rejected as invalid input.verified
- A JSON Web Token is a compact URL-safe means of representing claims encoded as a JSON object payload of a JSON Web Signature structure, represented as three base64url-encoded parts (header, payload, signature) separated by periods, enabling claims to be digitally signed or MACed.verified
- Organization role permissions are global within an organization so an editor can see and update all dashboards unless specifically restricted, and dashboard or folder permissions specified on an entity override the organization permissions for that entity.verified
- An RBAC permission comprises an action describing what task a user can perform and a scope describing where the action can be performed.verified
- Grafana RBAC offers basic roles (standard roles every user must have), fixed roles (discrete roles that cannot be changed or deleted and are assignable to users, teams, and service accounts), and custom roles (unique combinations of permission actions and scopes).verified
- Grafana RBAC scopes use wildcards: a folders wildcard matches any folder, a folders uid scope matches one folder, and a permission granted to a folder cascades down to subfolders located under it.verified
Cited sources
- Basic and fixed roles permissions · Grafana Labs
- Grafana Roles and permissions · Grafana Labs
- Grafana RBAC permission actions and scopes · Grafana Labs
- Folder access control · Grafana Labs
- RFC 7519: JSON Web Token (JWT) · IETF
- Role-based access control (RBAC) overview · Grafana Labs