Schema-on-read vs schema-on-write
The choice between checking data's shape before saving it and storing it raw so each reader interprets it later.
See it
What it is
Schema-on-write validates and converts data before it enters its main store; a PostgreSQL table rejecting text in an integer column is the everyday example. Schema-on-read keeps the raw bytes first and applies a shape when someone queries them, as with JSON files in an S3 data lake. Most real systems mix the two rather than choosing one forever.
Use schema-on-write when every reader needs dependable fields and bad data should fail at the boundary. Use schema-on-read when sources vary, the future questions are unknown, or preserving the original input matters. A common pattern is an immutable raw zone followed by a validated, typed curated zone, which Databricks names the medallion architecture and labels bronze, silver, and gold.
Gotcha: schema-on-read does not remove the schema; it makes every reader negotiate versions, missing fields, and malformed records. Schema-on-write can instead turn an uncoordinated producer change into rejected data. Keep raw inputs for replay, version contracts, quarantine failures, and measure how many records fail validation. A reader cannot reconstruct information that the producer never sent.
Ask AI for it
Build a two-zone event pipeline on Amazon S3. Land every original JSON payload unchanged in a date-partitioned raw prefix, then validate it with Ajv against a versioned JSON Schema Draft 2020-12 contract before writing typed Apache Parquet to a curated prefix. Put unknown schema versions and validation failures in a quarantine prefix with the error list, never discard the raw object, and publish counts for accepted, rejected, and unknown-version records. Include one backward-compatible schema evolution, a replay command, and consumer tests for missing, added, and mistyped fields.