Ship & Run

Observability & Logging

Knowing what your system is doing in production.

The territory

30 core terms mapped for this field, ranked by how often builders reach for them. Each one is a future entry. Want to bust one? One entry, one file, one pull request.

  • observabilityinferring internal system state from the telemetry a system emits"figuring out why it broke without redeploying" · "o11y"
  • telemetrysignals a system emits about its behavior, performance, and health"the data my app reports about itself" · "the numbers coming out of the app"
  • structured logginglogs emitted as machine-parseable JSON key-value fields, not prose"logs I can actually search" · "logs as objects instead of sentences"
  • log levelseverity tag (debug/info/warn/error) controlling what gets emitted"turn the noisy logs off in prod" · "only show me the bad ones"
  • error trackingservice that captures, groups, and alerts on runtime exceptions"tell me when users hit a crash" · "exception inbox"
  • correlation IDunique ID stitching all logs from one request together"trace one user's request across services" · "a tag that follows the request"
  • stack tracefunction call chain captured at the moment of an error"the wall of text showing where it broke" · "the crash printout"
  • metrictimestamped numeric measurement tracking system behavior over time"a number I can graph over time" · "the stat on the chart"
  • alert thresholdrule firing a notification when a metric crosses a limit"ping me if errors spike" · "notify me when it gets bad"
  • uptime monitoringexternal pings checking your site responds, from outside your infra"is my site down right now" · "something that checks the site every minute"
  • health check endpointURL returning OK so load balancers know the app lives"the /healthz thing" · "a page that says the server is alive"
  • latency percentileresponse time at a given rank, such as the slowest 5%"p95 / p99" · "not the average, the bad tail"
  • traceend-to-end record of one request, built from linked spans"the full journey of one request" · "the timeline for a single request"
  • distributed tracingfollowing one request end-to-end across every service it touches"see the whole path of a request" · "which service made it slow"
  • spanone timed operation inside a trace, nestable into a timeline"the individual bars in that timing chart" · "one step of the request"
  • OpenTelemetry (OTel)vendor-neutral standard for emitting traces, metrics, and logs"instrument once, switch tools later" · "the standard logging/tracing format"
  • instrumentationcode added to emit telemetry; auto (agent-injected) or manual"wiring the app up to report things" · "adding the tracking hooks"
  • log aggregationshipping logs from all machines into one searchable place"all my logs in one search box" · "central log pile"
  • live tailstreaming logs in real time as they happen"watch the logs scroll live" · "follow mode"
  • error groupingcollapsing thousands of identical errors into one deduplicated issue"fingerprinting" · "stop 5000 emails for the same bug"
  • samplingkeeping only a fraction of traces/logs to cut cost and volume"don't record every single request" · "keep 1 in 100"
  • SLO (service level objective)explicit target like "99.9% of requests succeed monthly""the reliability promise" · "how much downtime is acceptable"
  • error budgetallowed failure amount remaining before you breach your SLO"how much breakage I can still afford" · "downtime allowance"
  • golden signalslatency, traffic, errors, and saturation: the four charts always worth watching"the only four charts I need" · "the standard dashboard set"
  • high cardinalityfields with huge numbers of distinct values, like user_id"tagging by user blows up the bill" · "too many unique values"
  • RUM (real user monitoring)telemetry collected from actual browsers of actual visitors"real-world speed, not lab tests" · "what it's like on their phone"
  • APM (application performance monitoring)tooling that profiles code paths, queries, and endpoints in production"which endpoint and which query is slow" · "the perf x-ray"
  • heartbeat monitoring (dead man's switch)alert fires when a scheduled job fails to check in"tell me if my cron didn't run" · "alert on silence"
  • log retentionhow long logs are stored before deletion, often tiered hot/cold"how far back can I search" · "keep 30 days then archive"
  • PII scrubbing (log redaction)stripping emails, tokens, and card numbers before logs are stored"don't log passwords" · "mask the sensitive fields"

Deeper in the field

  • counter / gauge / histogram the three metric shapes: cumulative, point-in-time, distribution
  • time-series database storage optimized for timestamped metric points
  • cardinality explosion metric costs blowing up from too many label combinations
  • flame graph stacked-bar visual showing where CPU time is actually spent
  • continuous profiling always-on production profiler sampling CPU and memory
  • anomaly detection alerting on deviation from learned normal instead of fixed thresholds
  • trace context propagation passing traceparent headers so traces survive service hops
  • OpenTelemetry Collector agent that receives, processes, and routes telemetry
  • exporter component that sends collected telemetry to a specific backend
  • wide event one fat structured event carrying broad request context
  • canonical log line single summary log line emitted per request
  • downsampling (rollup) compressing old high-resolution metrics into coarser buckets
  • log-based metric counter derived by matching a pattern in log volume
  • service dependency map auto-generated graph of which services call which
  • synthetic monitoring scripted robot journeys run on a schedule against production
  • severity level (SEV) incident grading scale driving response urgency
  • MTTR / MTTD mean time to recover / detect, the incident speed metrics
  • burn-rate alert fires when error budget is being consumed too fast
  • alert fatigue responders ignoring alerts because too many are noise
  • blameless postmortem written incident review focused on systems, not individuals
  • black-box vs white-box monitoring probing from outside versus reading internal state
  • span attributes (tags) key-value metadata attached to a span for filtering
  • breadcrumbs trail of user actions recorded before an error, for context
  • log drain pipe forwarding a platform's logs to an external destination
  • on-call escalation policy rules deciding who gets paged, when, and who's next
  • runbook step-by-step doc for handling a specific alert or failure
  • SLI (service level indicator) measured reliability signal used to evaluate an SLO
  • alert rule evaluated condition triggering notifications or incident actions
  • dashboard saved visual view combining operational charts, tables, and queries
  • telemetry attribute key-value context used to filter, group, and aggregate telemetry
  • trace ID identifier shared by every span in one distributed trace
  • error rate proportion of operations failing within a defined time window
  • throughput (request rate) number of operations handled per unit of time
  • log parsing extracting searchable fields from unstructured log messages
  • symbolication converting stack addresses into readable functions and source locations