Trace sampling: head-based vs. tail-based

A service handling thousands of requests per second, each producing a trace with a dozen spans, generates an enormous volume of tracing data if every single request is fully recorded — real storage cost, and real overhead on the application itself to generate and export all of it. Sampling — recording only a subset of traces in full — is the standard answer, mirroring lesson 4's log-sampling discussion, but tracing's version has an extra wrinkle logging doesn't: when the sampling decision gets made changes what's actually possible to sample on.

Head-based sampling makes the keep-or-discard decision at the very beginning of a request — often a simple fixed percentage (sample 1% of all requests), decided before the request has actually done anything:

This is simple and cheap — the decision costs almost nothing, and every service downstream just needs to respect the decision already encoded in the trace context (lesson 6) as it propagates. The real weakness: the decision is made before knowing whether this particular request was actually interesting — a request that ends up being the one slow, or the one that errors out, has only a 1% chance of being sampled at all, exactly like any other request, even though it's precisely the kind of request worth having a full trace for.

Tail-based sampling buffers a request's full trace data temporarily, waits until the request actually completes, and then decides whether to keep it — based on what actually happened:

A common tail-based policy: keep 100% of traces for requests that errored or exceeded a latency threshold, and only a small sample of everything else. This directly solves head-based sampling's core weakness — the requests most worth investigating are exactly the ones guaranteed to be kept, not left to chance.

Making a decision only after a request completes means every span has to be buffered somewhere (typically in the collector, from lesson 6) until the outcome is known — a real memory and infrastructure cost that head-based sampling, which discards unsampled data immediately, doesn't pay. For a system with many, often long-lived or asynchronous requests, this buffering cost can be substantial, which is exactly why head-based sampling remains a completely reasonable default for simpler systems, and tail-based sampling is usually adopted specifically when the value of never missing an error trace outweighs the added infrastructure cost.

Whichever strategy is used, every service participating in one trace needs to agree on the same keep-or-discard decision — a trace where one service's spans were sampled in and another's were sampled out (because each service made its own independent, uncoordinated decision) produces an incomplete, misleading trace, the same broken-trace problem lesson 6 described for context propagation failures. Head-based sampling handles this naturally, since the decision is made once and carried in the propagated trace context every downstream service reads and respects. Tail-based sampling needs the collector layer itself to coordinate this consistently across all the spans it eventually receives for a given trace ID.