Data freshness / latency
The wait between something happening and its appearance in a report, including collection, processing, and dashboard cache time.
See it
What it is
Data latency is the delay between an action happening and the result becoming queryable in a report, and freshness is how recent the newest row in that report is. A click at 10:00 that appears at 10:15 has 15 minutes of end-to-end latency. Client batching, ingestion queues, warehouse loads, transformations, and dashboard caches can each own part of that gap.
Reach for a freshness measure when a decision has a deadline: incident response may need minutes, while a weekly finance report may tolerate hours. Track two clocks separately: ingestion delay from occurred_at to ingested_at, and report freshness from now back to the latest completed pipeline run. dbt encodes exactly this in its source freshness check, where each source declares warn_after and error_after.
Gotcha: the newest bucket is usually incomplete. Late events can revise yesterday's total after somebody already copied it into a slide. Put an 'updated at' timestamp and freshness status on the report, define when a period is final, and do not turn a partial hour into an anomaly alert.
Ask AI for it
Build a data freshness monitor for my PostgreSQL analytics pipeline. For each source, compute ingestion latency as ingested_at - occurred_at and current freshness as NOW() - MAX(ingested_at), reporting p50, p95, and maximum latency in UTC. Store an SLA per source, mark green, warning, or stale against that SLA, and show the last successful transformation timestamp on every dashboard. Send a Slack webhook only after two consecutive stale checks, and suppress anomaly detection for any report bucket whose source is stale.