E Explainarium

Observability System Design

Internal mechanics of custom distributed observability services: ingestion, storage, query, control plane, and platform services, taught at petabyte scale.

10 concepts, ordered as a guided path. Start at the top and follow the thread.

Start the path
0 / 10

Track 1 Ingestion

1 concept

High-throughput telemetry intake: backpressure, buffering, rate limiting, and noisy-tenant isolation.

  1. 1 High-Throughput Ingestion Pipeline expert An ingestion front for a petabyte-scale multi-tenant observability platform: backpressure, persistent buffering, per-tenant rate isolation, and the distributor-to-ingester write path.

Track 2 Storage

3 concepts

Compressed, immutable storage for metrics, logs, and traces: WAL, columnar blocks, and index sharding.

  1. 2 Time-Series Database Internals expert TSDB storage: delta-of-delta and XOR compression, WAL, object-storage block flushes, compaction, and cardinality-sharded inverted indexes.
  2. 3 Distributed Log Aggregation expert A log engine with label-driven ingestion, chunk streaming, and decoupled read and write paths.
  3. 4 Distributed Tracing Engine expert A trace store with head and tail sampling, a Parquet columnar schema, Bloom-filter trace lookup, and TraceQL planning.

Track 3 Query

2 concepts

Parallel query execution: splitting, pushdown, streaming results, and OOM mitigation.

  1. 5 Distributed Metrics Query Engine expert Parallel range-query execution with pushdown, sub-query splitting, and OOM-safe fan-out.
  2. 6 Real-Time Observability Backend expert Live telemetry pushed over WebSockets and SSE with query caching and partial progressive rendering.

Track 4 Control Plane

2 concepts

Acting on telemetry: rule evaluation, real-time views, and global synthetic probing.

  1. 7 Alerting and Rule Evaluation Engine expert Continuous rule evaluation over a consistent hash ring with grouped evaluation loops and distributed deduplication.
  2. 8 Global Synthetic Monitoring Infrastructure expert Worldwide probe orchestration, network-partition resilience, and real-time SLA/SLO aggregation.

Track 5 Platform

2 concepts

Cross-cutting services every stage depends on: authorization/RBAC and rate limiting.

  1. 9 Multi-Tenant Authz and RBAC Service expert Sub-millisecond authorization with hierarchical permission caching and gossip-spread eventual consistency.
  2. 10 Distributed Rate Limiter and Concurrency Controller expert Token and leaky bucket limits across a Redis cluster with race-free state synchronization.