Design Metrics & Monitoring

Assume 400,000 monitored hosts and containers, each exposing about 1,000 series, collected every 10 seconds. Container churn replaces a third of all series every day.

A small agent on each host scrapes local endpoints every 10 seconds, adds host labels and ships batches to an ingest gateway. Pushing from an agent rather than pulling from a central server means the platform does not need network reach into every subnet, and the agent can buffer through a short outage.

The gateway authenticates the tenant, enforces quotas and writes batches into Kafka, partitioned by a hash of the series labels so all samples for one series land in order on one partition. Ingesters consume those partitions and keep the most recent two hours of each series in memory as compressed chunks, writing a local write-ahead log for crash recovery. Every two hours, an ingester cuts its in-memory data into an immutable block and uploads it to object storage.

Queries go through a frontend that splits long ranges by day, serves repeated pieces from a cache, and fans out to two kinds of readers: ingesters for the most recent hours, and store gateways that read blocks from object storage for everything older. A compactor merges small blocks into large ones and produces downsampled versions. The rule evaluator is just another query client that runs every alert expression on a schedule and sends firing alerts to an alert manager, which deduplicates, groups and routes them.

Time series data is extremely regular, and the encoding exploits that. Timestamps for a series arrive at nearly fixed intervals, so store the first timestamp, then the first delta, then only the difference between consecutive deltas. For a perfectly steady 10-second scrape that difference is zero and costs a single bit. Small jitter fits in a handful of bits.

Values change slowly too. XOR each float with the previous one: identical values give zero (one bit), and similar values share their sign, exponent and leading mantissa bits, so you store only the meaningful middle bits plus a short header. This scheme, popularised by Facebook's Gorilla paper, brings a 16-byte sample to roughly 1.3 to 1.5 bytes on typical infrastructure data.

Samples are grouped into chunks of about 120 points per series. A chunk is the unit you compress, cache and read, which means a query for one hour of one series touches only a few chunks.

Every block carries an inverted index: for each label pair such as service="checkout", a sorted list of the series IDs that have it. A query selector is a set of label matchers, so the engine fetches each matching postings list and intersects them. Because the lists are sorted, intersection is a linear merge. Regex matchers expand to the union of the matching label values first. The index also stores label names and values on their own, which is what dashboard dropdowns query.

The most expensive pattern is a matcher that hits millions of series, such as every pod of every service. Query planning should apply the most selective matcher first and refuse queries that would scan more than a set limit of series, returning a clear error instead of dragging the cluster down.

Each new label value creates a new series, and each series costs memory for its head chunk and index entry whether it receives one sample or a million. A single developer who adds user_id as a label to a request counter can create tens of millions of series in an hour. This is the most common way metrics platforms fall over.

Defend in layers. Per-tenant limits on active series, enforced at the gateway, reject new series once a tenant passes its quota while still accepting samples for existing ones. Per-metric limits catch a single runaway metric. Relabeling rules in the agent drop known bad labels before they leave the host. And a cardinality explorer shows teams which metrics and labels cost the most, so they can fix the cause.

A 90-day graph at 10-second resolution would need almost 780,000 points per series, far more than a screen can show. The compactor writes rollups at 5 minutes and 1 hour, storing count, sum, min, max and last per window so that averages, rates and extremes can be computed correctly. The query engine picks the coarsest resolution that still gives enough points for the requested step.

Rules are sharded across evaluators by hash, and each one runs its rules on a fixed interval. A rule with a for clause only fires after the condition has been true for that long, which removes flapping from single bad scrapes. The evaluator also needs to alert on absence: a rule like "this service reported nothing for five minutes" catches a dead agent that would otherwise look like a quiet, healthy system.