Design Distributed Logging Framework

On one server, debugging is ssh and grep. The log file is local, it sits next to the process that wrote it, and the timeline of one request is a few consecutive lines. Move to 1,500 services running as 60,000 short-lived containers and all three of those properties vanish. Containers are rescheduled and their disks disappear with them. One checkout request touches the gateway, the cart, pricing, inventory, payments and an email sender, each on a different node. By the time someone notices a problem, the pod that logged the clue may have been gone for an hour.

We need a central system that collects every log line from every container, stores it durably and cheaply, and lets engineers ask questions such as "show me every ERROR from the payments service in the last 30 minutes that mentions this order ID". It also needs a live view for watching a deploy, and alert rules that fire when error patterns appear. Think of the internal platform behind tools like Grafana Loki, Elastic, Splunk or Datadog Logs.

Start from the fleet: 1,500 services averaging 40 pods each, every pod emitting about 6 lines per second. Average line size including labels is 380 bytes. Queries are counted per engineer: about 4,000 engineers, a handful of searches each on a normal day.

Three audiences call the platform: agents writing, people searching, and people watching. Each gets an interface shaped for it.

The pipeline has one rule: every stage can fall behind without hurting the stage before it. The agent buffers to local disk, the gateway acknowledges once Kafka has the batch, and ingesters, tail and alerts read from Kafka at their own pace. A slow indexer delays search; it never makes a checkout service block on a log write.

The agent runs as a daemon on each node, something like Fluent Bit or Vector. It tails container log files, parses JSON, adds Kubernetes labels, batches by size or time, compresses with zstd, and posts to the gateway. It keeps up to 2 GB of unsent data on disk, which covers about an hour of a busy node, and it numbers each batch so the gateway can drop repeats after a retry.

The gateway authenticates the cluster, applies per-tenant rate limits, rejects label sets that would explode cardinality, and writes to Kafka keyed by stream, where a stream is one unique label set. Keying by stream keeps one pod's lines in order on one partition. Kafka holds 24 hours of data, about 12 TB raw, so the downstream stages can be redeployed or rebuilt without data loss.

Ingesters consume partitions and accumulate lines per stream in memory. When a stream's buffer reaches about 1.5 MB compressed or 15 minutes of age, the ingester sorts it by time, compresses it into a chunk, builds a small bloom filter of the tokens in it, writes both to object storage, and records the chunk in the label index. Recent chunks also stay on local SSD for the hot tier. Until a chunk is flushed, queriers ask the ingesters directly for the newest lines, which is how search stays within 10 seconds of real time.

The query path splits a request into time slices, looks up candidate chunks in the label index, drops chunks whose bloom filter rules out the search term, and fans the remaining chunks out across a pool of stateless queriers that decompress and scan them in parallel. Results merge by timestamp and stream back to the UI.

A relational table with one row per line dies on write volume and index maintenance long before a year of data is in it, and a LIKE '%timeout%' query cannot use a B-tree index at all. A general search engine with an inverted index over every token handles queries beautifully, but every line is tokenized, every token appended to posting lists, and segments are merged in the background, all on SSD with replicas. At 360,000 lines a second you end up running a very large cluster mostly to index text that will never be searched.

The chosen layout treats a log stream like a time series of text. Lines are grouped by stream and time into chunks of a few megabytes, compressed together, which works well because neighbouring lines from one pod are very repetitive. The only index is over labels and time: which chunks belong to service=charge-api, level=error between 08:00 and 09:00. That index is tiny because there are perhaps a few million active streams, not trillions of words.

Free-text search then becomes a parallel scan of the chunks that survive the label filter. Two tricks make that scan cheap. First, each chunk carries a bloom filter of its tokens, around 1% of its size, so a search for a rare string like an order ID can skip most chunks without downloading them. Second, the scan itself is embarrassingly parallel: a querier can decompress and grep about 1 GB of raw text per core per second, so 200 querier cores read a 30-minute slice of a large service in well under a second.

This design depends on labels describing where a line came from, not what it says. If a team puts a user ID or request ID in a label, every user becomes its own stream, chunks shrink to a handful of lines, the index balloons and compression collapses. The gateway enforces limits, such as at most 15 labels per stream and at most 50,000 active streams per service, and rejects or folds offenders with a clear error. High-cardinality values belong inside the line, where the bloom filter and the scan can find them.

Pure scanning is fine for needle searches but slow for questions like "p95 latency by endpoint over a week". For those, a small set of well-known JSON fields such as status, latency_ms and route can be extracted at ingest into a columnar side table in Parquet or ClickHouse. That gives fast aggregations for the 10 fields everyone uses without paying to index the other 500.