Sampling
Keeping only a fraction of your traces and logs, say 1 in 100, so the volume and the bill stay sane without losing the picture.
See it
What it is
Recording every trace and every log line from a busy service costs more than running the service. Sampling keeps a fraction (1 in 10, 1 in 1,000) and drops the rest, betting that a representative slice tells you the same story as the whole thing. It usually does, right up until the one request you needed was in the dropped 99%.
Two flavors matter. Head-based decides at the start of a trace, before anything interesting has happened: cheap, stateless, and it must propagate the decision to every downstream service so you never get half a trace. Tail-based buffers the whole trace in a collector, then decides after the fact, so you can keep 100% of errors and slow requests while sampling boring 200s at 1%. Neither one simply wins. Head-based stays cheap at volumes where buffering every trace is not an option, which is why large systems still choose it; tail-based buys you the interesting traces and charges you memory, a delay before the decision, and a collector tier to run it in.
Gotcha: sampled data does not give you exact totals. A sample taken at a known probability can still estimate counts, if you record the effective sample rate as an attribute and weight by its inverse, but what you get back is an estimate with error bars rather than the real number, and the estimate degrades badly for rare events. It breaks entirely once tail sampling makes the keep decision depend on the data itself, since errors are then wildly over-represented on purpose. Treat unsampled metrics as the authoritative counters, and traces as the shape of things.
Ask AI for it
Set up trace sampling for this service with OpenTelemetry, and put the keep-or-drop decision in exactly one place. In the SDK, configure a consistent AlwaysOn sampler in every participating service so all spans are recorded and exported and no trace arrives half recorded. Do not drop anything before the Collector, because a head-based decision is made before the request has had a chance to fail. Then add an OpenTelemetry Collector with a tail_sampling processor as the only sampler: keep 100% of traces containing an error status, 100% of traces slower than 1s, and 5% of everything else. Record the effective sample rate as a span attribute so counts can be weighted back up, and keep error-level logs and all metrics unsampled so authoritative counts never depend on the sample.