Distributed tracing
Following one request as it hops through every service, so you can see which hop actually made it slow.
See it
What it is
Distributed tracing gives one request a shared trace ID, then every service it touches records what it did as a span stamped with that ID. Stitch the spans back together and you get a waterfall: gateway 4ms, auth 12ms, product service 30ms, and one N+1 query loop burning 900ms. Logs tell you that something happened somewhere. A trace tells you where, in what order, and how long each hop took.
Reach for it the moment a request crosses a process boundary: microservices, serverless functions calling each other, a frontend hitting an API that hits a database and two vendors. The plumbing is context propagation: a traceparent header (the W3C Trace Context standard) rides along every outbound call so the next service knows which trace it belongs to. OpenTelemetry auto-instrumentation covers most HTTP clients, servers, and database drivers for free.
Gotcha: traces break silently at the seams your library did not patch. A queue, a background job, or a hand-rolled fetch wrapper that drops the traceparent header leaves you with two disconnected half-traces and no error telling you so. Second gotcha: capturing 100% of real traffic is a bill, so most teams sample, and head-based sampling throws dice before it knows the request was the slow one. Tail-based sampling (decide after the trace finishes) keeps the errors and the stragglers.
Ask AI for it
Instrument this service with OpenTelemetry distributed tracing. Install the OTel SDK plus auto-instrumentation for the HTTP server, the HTTP client, and the database driver, export over OTLP to the endpoint in an env var, and set service.name, service.version, and deployment.environment as resource attributes. Propagate W3C traceparent headers on every outbound call, including background jobs and queue messages, so traces survive async hops. Add manual spans around the two or three slowest business operations with useful attributes (route, operation, retry count, cache hit), record exceptions on the span and set error status, and keep user emails and tokens out of attributes. Use tail-based sampling that keeps 100% of errors and slow traces and 5% of the rest.