What Is Distributed Tracing? A Practical Guide for Developers

A checkout request reaches the orders service, which calls payment, inventory, and shipping. The page takes six seconds to return.

Individual service logs can show that each component handled a request. Distributed tracing shows how those operations belong to the same execution path and where the time was spent.

POST /checkout
├── Orders service       45 ms
├── Payment service     220 ms
├── Inventory service    90 ms
└── Shipping API       5200 ms

The trace attributes most of the request time to the shipping call.

A Trace Is Made of Spans

A trace represents an execution path through a system. It contains spans, where each span records one operation.

Trace
├── HTTP request
├── database query
├── payment call
└── shipping call

A span commonly records a name, start and end timestamps, attributes, status, events, and its relationship to other spans.

The trace ID groups spans that belong to the same trace. Each span has its own span ID.

trace_id = 4bf92f3577b34da6a3ce929d0e0e4736
span_id  = 00f067aa0ba902b7

OpenTelemetry represents trace IDs as 16 bytes and span IDs as 8 bytes.

Trace Context Has to Cross Service Boundaries

Tracing breaks when the receiving service cannot connect its span to the caller’s trace.

For HTTP, W3C Trace Context defines the traceparent and tracestate headers. A traceparent value carries the trace ID, the caller’s parent ID, and trace flags.

traceparent:
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Instrumentation extracts that context on the receiving side, creates the next span with the appropriate parent relationship, and injects updated context into outgoing calls.

Other transports need equivalent propagation. Message queues, RPC systems, and background jobs are common places for traces to become disconnected when context is not carried forward.

Parent-Child Relationships Build the Trace

A simple synchronous call chain can produce a tree:

POST /checkout
└── Orders service
    ├── SQL: insert order
    ├── Payment service
    └── Inventory service

The timestamps show duration. Parent and span identifiers show how operations relate.

Not every distributed workflow forms a simple tree. OpenTelemetry spans can also contain links to causally related spans, which is useful for messaging and other asynchronous work where a strict parent-child relationship may not describe the operation well.

Instrumentation Creates the Telemetry

Applications need instrumentation around the operations worth tracing.

Some libraries and frameworks can be instrumented automatically. Custom business operations may need manual spans:

HTTP server span
   ↓
checkout span
   ├── payment client span
   └── inventory client span

OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions, and exporters for collecting telemetry and sending it to a backend.

Instrumentation quality matters. A trace with missing service boundaries or meaningless span names can be difficult to use even when trace propagation works correctly.

Traces Answer Request-Level Questions

Metrics are useful for aggregate questions:

What is the p95 checkout latency?
How many requests failed?
Did error rate increase?

A trace is useful when the next question is:

Why did this checkout take 6 seconds?
Which dependency failed for this request?
What calls occurred before the error?

Logs provide event detail. Traces provide execution relationships. Metrics provide aggregate behaviour.

The three signals are often used together during an incident rather than as substitutes for one another.

Sampling Controls Trace Volume

Recording and retaining every trace can be expensive in a high-traffic system.

Sampling limits the amount of telemetry kept or exported. A system might retain a fraction of normal traffic while applying different policies to errors, high-latency requests, or selected services.

The sampling strategy affects what engineers can investigate later. If a trace was not retained, the backend cannot display it simply because an incident was discovered afterward.

W3C trace flags include a sampled bit that propagates a sampling decision, while tracing systems can implement additional sampling behaviour.

Choose Span Attributes Carefully

Span attributes help engineers filter and understand traces:

http.request.method = POST
http.response.status_code = 503
service.name = shipping-client
order.id = 847291

Useful attributes describe the operation. Large payloads, secrets, access tokens, passwords, and unnecessary personal data should not be attached to spans.

High-cardinality attributes can also increase storage and indexing cost, although tracing backends handle cardinality differently.

Correlate Traces With Logs and Business Identifiers

A trace ID in a log entry lets an engineer move from an error message to the surrounding trace.

{
  "level": "error",
  "message": "Shipping request timed out",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "order_id": "847291"
}

The trace ID identifies the execution path. An order_id or other correlation value can group work that lasts across multiple traces.

That distinction matters for asynchronous and long-running business processes.

Tracing Has Operational Failure Modes

Several implementation mistakes reduce the value of tracing:

context is not propagated through a queue
services use inconsistent span names
important errors are not recorded
attributes contain secrets
sampling removes the traces needed for diagnosis
instrumentation creates huge numbers of low-value spans

Trace collection also depends on exporters, collectors, and the backend. Telemetry should not become a new critical dependency for the application request path; instrumentation needs bounded queues, timeouts, and failure behaviour appropriate to the system.

When Distributed Tracing Becomes Useful

A small application with one process and a database may be easy to debug with logs and metrics alone.

Tracing becomes more useful as one user operation crosses service, process, network, queue, or third-party boundaries. Those boundaries are where timing and causality become harder to reconstruct from isolated telemetry.

The practical test is whether engineers routinely need to answer where a particular request went and which operation consumed its time. Distributed tracing records the relationships needed to answer that question.