Distributed tracing and OpenTelemetry

A span represents one unit of work — one function call, one database query, one HTTP request to another service — with a start time, a duration, and a set of key-value attributes describing it. A trace is simply a tree of spans, connected by parent-child relationships, that together represent one complete request's journey:

Each span is created independently, by whatever code is executing at that moment — a trace isn't recorded as one object, it's assembled afterward from many independently-reported spans that all share the same trace ID.

For spans created in different services (or even different threads within one service) to end up correctly nested in the same trace, each one needs to know the current trace ID and its own parent span ID — information that has to travel with the request itself, typically as HTTP headers:

This is standardized as the W3C Trace Context specification — precisely so that a request passing through services written by different teams, in different languages, using different tracing libraries, still produces one coherent trace instead of several disconnected fragments that happen to describe the same request. Without propagating this header correctly across every service boundary, tracing silently breaks into unconnected pieces — a genuinely common integration bug when adding tracing to an existing system.

Before OpenTelemetry (OTel), adopting a specific tracing vendor (X-Ray, Datadog, custom-built) meant instrumenting application code against that vendor's specific SDK — switching vendors later meant re-instrumenting everything. OpenTelemetry standardizes the instrumentation layer itself, independent of where the data ultimately goes:

Application code is instrumented once, against OTel's vendor-neutral API. An OTel Collector receives that telemetry and exports it to whichever backend(s) are actually configured — switching from one observability vendor to another becomes a collector configuration change, not a rewrite of instrumentation scattered across every service.

Most OTel language SDKs include auto-instrumentation for common libraries (HTTP frameworks, database drivers, message queue clients) — spans for these get created automatically, with no code changes needed, covering the majority of a typical trace's structure. Manual instrumentation — explicitly starting a span around a specific block of business logic — fills in the gaps auto-instrumentation can't know about: a specific, meaningful step in a checkout flow that isn't itself a library call, but is exactly the kind of thing worth seeing broken out in a trace when investigating why checkout was slow.

A trace is exactly the tool lesson 1 described for the "where did the time go" question — but only if trace context actually propagates correctly across every service boundary a request crosses. A trace that silently stops at a service boundary because that service's client library wasn't instrumented, or didn't forward the traceparent header, looks complete but is actually missing everything downstream — a subtle, easy-to-miss gap that makes an incomplete trace look deceptively trustworthy.