Distributed Tracing: Following a Slow Request Across Services

Key takeaways

• A trace connects recorded operations through identifiers, timing, and relationships.
• Parent and child durations overlap, so adding span durations can misrepresent request latency.
• Missing propagation, instrumentation, or exported data can leave an incomplete account of an execution.
• Sampling determines which traces remain available and affects what conclusions can be drawn from them.

Checkout takes six seconds, but the orders service reports that its database write finished quickly. The payment service also appears healthy. Somewhere between the incoming request and the response, the customer is waiting.

Distributed tracing helps connect the operations belonging to that execution. Instead of searching several service logs by approximate time, an engineer can inspect a related set of recorded operations and identify which part deserves closer examination.

That account is valuable, but it is not automatically complete. The usefulness of a trace depends on what was instrumented, how context travelled, and which records reached the backend.

Read the Trace as Related Operations

In OpenTelemetry’s terminology, a span records an operation with timing and contextual information. Spans can have attributes, events, status, and relationships to other spans. A trace groups related spans under a trace ID.

For an illustrative checkout, the recorded durations might be:

Operation	Duration	Relationship
Checkout request	6,000 ms	Root operation
Save order	45 ms	Child of checkout
Authorise payment	220 ms	Child of checkout
Obtain shipping quote	5,200 ms	Child of checkout

The shipping call is an obvious place to investigate. Its client span records how long the caller waited around that operation. Without downstream instrumentation, it does not tell us how much time the shipping provider spent computing, waiting on its own dependencies, or transferring the response.

Follow the Timeline Before Adding Numbers

A parent’s duration normally includes time spent waiting for its children. Adding the checkout duration to its child durations would therefore count some time twice. Parallel calls introduce another overlap: two one-second operations started together need not add two seconds to the request.

Inspect start times, end times, and the relationships between operations. Ask which chain of work determines when the response can be returned. A long background operation that finishes after the response may be important without explaining the customer’s wait.

A gap between recorded spans needs investigation too. It could represent uninstrumented work, a wait for capacity, or missing telemetry. Treat the gap as an unanswered question rather than assigning it a cause from the picture alone.

Establish Context at Each Boundary

OpenTelemetry’s context propagation guidance explains how instrumentation carries context between operations and services. For HTTP, propagators commonly use the W3C headers. Other transports require supported mechanisms to inject and extract the relevant context.

Consider checkout publishing a shipping job. If the producer records a span but the consumer never extracts the message context, the backend may display disconnected executions. The worker can be functioning correctly while the diagnostic relationship is missing.

Begin with one representative path through the application. Inspect the incoming request, a downstream call, and any queue boundary. Background tasks need particular attention because they may execute after the original request’s context has ended. Trace and correlation identifiers can connect different scopes without assuming that every business process must fit inside one trace.

Instrument the Decisions You Need to Understand

Automatic instrumentation can cover common server frameworks, clients, and database libraries. Business operations may need additional spans where a meaningful boundary is otherwise invisible.

For checkout, recording a shipping quote operation may help distinguish provider lookup from local price calculation. Creating a span for every small helper function could add volume without making the request easier to explain. Choose names that describe stable operations; keep individual order identifiers out of span names.

Attributes should answer likely investigation questions. A shipping provider name, retry count, or response status may help compare requests. Full customer addresses and request bodies are unlikely to be necessary for understanding a timeout. In OpenTelemetry, service identity such as service.name is generally supplied as resource metadata describing the telemetry producer.

Make Sampling an Explicit Trade-Off

OpenTelemetry distinguishes head and tail sampling. Head sampling decides early, before the full outcome is known. Tail sampling makes a later decision using collected trace information, which can support policies favouring slow or failed executions.

Tail sampling needs infrastructure to buffer and process the incoming data. It also cannot recover spans discarded before they reached that decision point. A combined policy must account for that limitation.

For the slow checkout, a missing trace may mean it was never retained. It does not establish that the request never occurred. An error-focused sample also cannot be treated as an unbiased measure of the service’s overall failure rate. Use aggregate metrics designed for that question, and understand whether the telemetry pipeline applies weighting or other corrections.

Move Between Signals With a Question

Metrics can show that checkout latency increased after a deployment. A retained trace can identify a particular slow execution. Logs associated with its trace and span IDs can add details about a retry or provider response.

Use those signals to test a specific explanation. If the theory is that a new release added retries, compare the recorded attempts and timing before and after the change. If the provider is slow on the first attempt, retry count alone will not explain the original delay.

Correlation IDs and trace IDs help connect those observations across services so you can test a hypothesis against the same request’s events. A visually prominent span is a lead, not a complete causal explanation.

Operate the Telemetry Path Too

Exporters, collectors, and storage can fail or fall behind. Set appropriate buffering and time limits, and make dropped telemetry visible. The application should have a deliberate response when its telemetry destination is unavailable, rather than accumulating unbounded diagnostic work.

A useful tracing setup earns its cost during an investigation. For checkout, it should help identify the affected dependency, distinguish attempts, and connect the relevant evidence. The outcome is a narrower, better-supported explanation of the delay, with missing information acknowledged rather than hidden by an apparently complete diagram.

A trace narrows the investigation by showing where recorded work happened; explaining why still requires evidence.