Data pipeline
A staged flow that pulls data out of one system, cleans and reshapes it, and loads it into another. The nightly job behind every dashboard.
See it
What it is
A data pipeline is a staged flow that pulls data out of source systems, reshapes it, and lands it somewhere useful. The classic spelling is ETL (extract, transform, load); the modern default is ELT, where you dump raw data into a warehouse first and transform it there with SQL, because storage is cheap and raw data you already loaded can be reprocessed. Batch pipelines run on a schedule; streaming pipelines process records as they arrive.
Reach for one when your dashboard needs data from four places at once, when analytics queries are slowing down the production database, or when some vendor's API is the only home for numbers you need. Typical stack: Fivetran or Airbyte to extract, a warehouse like BigQuery, Snowflake, or DuckDB to hold it, dbt to transform, and Airflow, Dagster, or a plain cron job to run the whole thing.
Gotcha: pipelines fail quietly and you find out from a stakeholder asking why the chart flatlined on Tuesday. Make every stage idempotent so a rerun of the same window produces the same result instead of duplicate rows, track an incremental watermark instead of reloading everything, and alert on freshness (data older than X hours) rather than only on crashes. The other silent killer is schema drift: an upstream API renames a field and your transform keeps happily writing nulls.
Ask AI for it
Build an incremental ELT pipeline that syncs the orders and customers tables into the warehouse nightly. Track the checkpoint as a composite (updated_at, primary key) cursor, not a bare timestamp, so rows sharing a timestamp cannot be skipped at the boundary. Each run: capture an upper bound first, select rows greater than the checkpoint and up to that bound with a small overlap window subtracted from the lower bound to catch late commits and out-of-order updates, load them raw into a staging schema, then transform with SQL into a clean analytics table. Upsert on primary key so the overlap and any rerun are idempotent rather than duplicating rows, and advance the stored checkpoint only after the load succeeds. Handle source deletions explicitly: propagate tombstones (a soft-delete flag the query can see) or switch that table to change data capture, because an updated_at cursor can never see a hard-deleted row. Wrap the load in a transaction, record run start, end, cursor bounds, and row counts in a runs table, and fail loudly with an alert if the newest row is more than 26 hours old.