Chunking
Cutting documents into retrievable pieces before embedding them. Too big and hits go vague; too small and the answer loses the thread.
See it
What it is
Retrieval returns chunks, not documents, so chunking decides what the model can ever see. The unit you want is a passage that stands on its own: one section, one procedure, one FAQ answer. Split on structure first (headings, list items, table rows, code fences), then fall back to a size cap, commonly 300 to 800 tokens with 10 to 20 percent overlap so a sentence straddling a boundary survives intact somewhere.
Better than tuning the number: keep the context that makes a chunk legible. Prepend the document title and heading path to every chunk, keep tables and code blocks whole, and store neighbouring chunk ids so a hit can be expanded into its surroundings before answering. Then evaluate against real questions, because chunking is the one retrieval knob you cannot reason your way to.
Gotcha: a fixed character split cuts mid-sentence and orphans pronouns, so you retrieve 'It must be renewed within 30 days' with no idea what 'it' is. Second gotcha: rechunking means re-embedding and reindexing everything, so keep the raw source files and treat chunk output as disposable.
Ask AI for it
Write a chunker for my markdown and PDF docs. Split on heading boundaries first, then pack sections into chunks of at most 600 tokens with 80 tokens of overlap, keeping code fences, tables and list items whole whenever the block fits under the cap. When a single structural block is bigger than the cap, split it with a strategy for its type (code by top-level statement, tables by row groups with the header row repeated, prose by sentence) and mark the resulting chunks as fragments of one block so downstream code knows they belong together. Prepend the document title and full heading path to each chunk's text. Emit chunk id, document id, ordinal, heading path, token count, the fragment flag and previous/next ids so a hit can be expanded to its neighbours. Make size and overlap config values, and include tests for a code block, a table and a single section that each exceed the cap.