Ship & Run
Observability & Logging
Knowing what your system is doing in production.
The territory
30 core terms mapped for this field, ranked by how often builders reach for them. Each one is a future entry. Want to bust one? One entry, one file, one pull request.
- observabilityinferring internal system state from the telemetry a system emits"figuring out why it broke without redeploying" · "o11y"
- telemetrysignals a system emits about its behavior, performance, and health"the data my app reports about itself" · "the numbers coming out of the app"
- structured logginglogs emitted as machine-parseable JSON key-value fields, not prose"logs I can actually search" · "logs as objects instead of sentences"
- log levelseverity tag (debug/info/warn/error) controlling what gets emitted"turn the noisy logs off in prod" · "only show me the bad ones"
- error trackingservice that captures, groups, and alerts on runtime exceptions"tell me when users hit a crash" · "exception inbox"
- correlation IDunique ID stitching all logs from one request together"trace one user's request across services" · "a tag that follows the request"
- stack tracefunction call chain captured at the moment of an error"the wall of text showing where it broke" · "the crash printout"
- metrictimestamped numeric measurement tracking system behavior over time"a number I can graph over time" · "the stat on the chart"
- alert thresholdrule firing a notification when a metric crosses a limit"ping me if errors spike" · "notify me when it gets bad"
- uptime monitoringexternal pings checking your site responds, from outside your infra"is my site down right now" · "something that checks the site every minute"
- health check endpointURL returning OK so load balancers know the app lives"the /healthz thing" · "a page that says the server is alive"
- latency percentileresponse time at a given rank, such as the slowest 5%"p95 / p99" · "not the average, the bad tail"
- traceend-to-end record of one request, built from linked spans"the full journey of one request" · "the timeline for a single request"
- distributed tracingfollowing one request end-to-end across every service it touches"see the whole path of a request" · "which service made it slow"
- spanone timed operation inside a trace, nestable into a timeline"the individual bars in that timing chart" · "one step of the request"
- OpenTelemetry (OTel)vendor-neutral standard for emitting traces, metrics, and logs"instrument once, switch tools later" · "the standard logging/tracing format"
- instrumentationcode added to emit telemetry; auto (agent-injected) or manual"wiring the app up to report things" · "adding the tracking hooks"
- log aggregationshipping logs from all machines into one searchable place"all my logs in one search box" · "central log pile"
- live tailstreaming logs in real time as they happen"watch the logs scroll live" · "follow mode"
- error groupingcollapsing thousands of identical errors into one deduplicated issue"fingerprinting" · "stop 5000 emails for the same bug"
- samplingkeeping only a fraction of traces/logs to cut cost and volume"don't record every single request" · "keep 1 in 100"
- SLO (service level objective)explicit target like "99.9% of requests succeed monthly""the reliability promise" · "how much downtime is acceptable"
- error budgetallowed failure amount remaining before you breach your SLO"how much breakage I can still afford" · "downtime allowance"
- golden signalslatency, traffic, errors, and saturation: the four charts always worth watching"the only four charts I need" · "the standard dashboard set"
- high cardinalityfields with huge numbers of distinct values, like user_id"tagging by user blows up the bill" · "too many unique values"
- RUM (real user monitoring)telemetry collected from actual browsers of actual visitors"real-world speed, not lab tests" · "what it's like on their phone"
- APM (application performance monitoring)tooling that profiles code paths, queries, and endpoints in production"which endpoint and which query is slow" · "the perf x-ray"
- heartbeat monitoring (dead man's switch)alert fires when a scheduled job fails to check in"tell me if my cron didn't run" · "alert on silence"
- log retentionhow long logs are stored before deletion, often tiered hot/cold"how far back can I search" · "keep 30 days then archive"
- PII scrubbing (log redaction)stripping emails, tokens, and card numbers before logs are stored"don't log passwords" · "mask the sensitive fields"
Deeper in the field
- counter / gauge / histogram the three metric shapes: cumulative, point-in-time, distribution
- time-series database storage optimized for timestamped metric points
- cardinality explosion metric costs blowing up from too many label combinations
- flame graph stacked-bar visual showing where CPU time is actually spent
- continuous profiling always-on production profiler sampling CPU and memory
- anomaly detection alerting on deviation from learned normal instead of fixed thresholds
- trace context propagation passing traceparent headers so traces survive service hops
- OpenTelemetry Collector agent that receives, processes, and routes telemetry
- exporter component that sends collected telemetry to a specific backend
- wide event one fat structured event carrying broad request context
- canonical log line single summary log line emitted per request
- downsampling (rollup) compressing old high-resolution metrics into coarser buckets
- log-based metric counter derived by matching a pattern in log volume
- service dependency map auto-generated graph of which services call which
- synthetic monitoring scripted robot journeys run on a schedule against production
- severity level (SEV) incident grading scale driving response urgency
- MTTR / MTTD mean time to recover / detect, the incident speed metrics
- burn-rate alert fires when error budget is being consumed too fast
- alert fatigue responders ignoring alerts because too many are noise
- blameless postmortem written incident review focused on systems, not individuals
- black-box vs white-box monitoring probing from outside versus reading internal state
- span attributes (tags) key-value metadata attached to a span for filtering
- breadcrumbs trail of user actions recorded before an error, for context
- log drain pipe forwarding a platform's logs to an external destination
- on-call escalation policy rules deciding who gets paged, when, and who's next
- runbook step-by-step doc for handling a specific alert or failure
- SLI (service level indicator) measured reliability signal used to evaluate an SLO
- alert rule evaluated condition triggering notifications or incident actions
- dashboard saved visual view combining operational charts, tables, and queries
- telemetry attribute key-value context used to filter, group, and aggregate telemetry
- trace ID identifier shared by every span in one distributed trace
- error rate proportion of operations failing within a defined time window
- throughput (request rate) number of operations handled per unit of time
- log parsing extracting searchable fields from unstructured log messages
- symbolication converting stack addresses into readable functions and source locations