Sampling
Keeping only a fraction of your traces and logs, say 1 in 100, so the volume and the bill stay sane without losing the picture.
See it
What it is
Recording every trace and every log line from a busy service costs more than running the service. Sampling keeps a fraction (1 in 10, 1 in 1,000) and drops the rest, betting that a representative slice tells you the same story as the whole thing. It usually does, right up until the one request you needed was in the dropped 99%.
Two flavors matter. Head-based decides at the start of a trace, before anything interesting has happened: cheap, stateless, and it must propagate the decision to every downstream service so you never get half a trace. Tail-based buffers the whole trace in a collector, then decides after the fact, so you can keep 100% of errors and slow requests while sampling boring 200s at 1%. Tail-based is what you want; it costs memory and a collector tier to get it.
Gotcha: sampled data cannot be counted. If you sample at 1% and then count errors from traces, your error count is off by 100x unless you record the sample rate as an attribute and weight by its inverse. Keep counting on unsampled metrics, and keep traces for the shape of things.
Ask AI for it
Set up trace sampling for this service with OpenTelemetry. Use a ParentBased TraceIdRatio sampler at 5% in the SDK so the sampling decision propagates via traceparent headers and no trace ends up half recorded. Then add an OpenTelemetry Collector with a tail_sampling processor that keeps 100% of traces containing an error status, 100% of traces slower than 1s, and 5% of everything else. Record the effective sample rate as a span attribute so counts can be weighted back up, and keep error-level logs and all metrics unsampled.