Quantization
Storing model weights with fewer bits so they use less memory and can run faster locally, at the cost of some numerical precision.
See it
What it is
Quantization stores a model's weights with lower-precision numbers, such as 8-bit or 4-bit values instead of 16-bit floating point. The model keeps the same architecture and parameter count, but its weight file and memory traffic shrink. With matching kernels, that can make local inference cheaper and faster enough to fit a model on hardware that could not hold the original.
Reach for post-training quantization when serving or running an existing model is limited by RAM, VRAM, download size, or memory bandwidth. llama.cpp's GGUF ecosystem makes the trade-off concrete: one base model can be published in several quantization levels so users choose a size their machine can handle. TheBloke's Hugging Face repos made that shelf famous in 2023: one model, a column of files from Q2 to Q8, and a table telling you which one your RAM could hold.
Gotcha: fewer bits trade away numerical detail, and the smallest file is not automatically the best deployment. Quality loss depends on the model, the quantization method, and the task; speed depends on hardware support. The context's KV cache also consumes memory separately from the weights, so quantizing a model does not make long contexts free.
Ask AI for it
Add a reproducible llama.cpp quantization pipeline for this F16 GGUF model. Use `llama-quantize` to produce a `Q4_K_M` GGUF artifact, retain the original F16 file as the baseline, and record the exact llama.cpp commit and command in a manifest. Benchmark both artifacts with `llama-bench` on the same hardware, prompt, context size, thread count, and batch size. Report file size, peak RAM or VRAM, prompt-processing speed, generation speed, and quality on the existing eval set; reject the quantized artifact if it misses the project's accuracy threshold.