Open-weights model
A model whose learned parameter files you can download and run yourself. The license, training code, and training data may still be closed.
See it
What it is
An open-weights model publishes the learned parameter files needed to run inference yourself. You can download them, serve them on your own hardware, and often quantize or fine-tune them. That does not automatically make the model open source: its training data may be private, its training code may be missing, and its license may restrict commercial use or redistribution. Llama 2 made that gap plain in 2023: the weights were a free download, and the license still barred services above 700 million monthly users.
Reach for one when data must stay on your machine, the product must work offline, request volume makes self-hosting economical, or you need control over adapters and inference settings. Hugging Face is the common distribution shelf, GGUF is a common file format for local quantized models, and llama.cpp is a common runtime for serving them.
Gotcha: downloadable does not mean free to operate. Memory, throughput, batching, monitoring, security patches, and model upgrades become your job. Quantization cuts memory use but can reduce quality, and every model carries its own license, so check that license before building a product around the files.
Ask AI for it
Package a supplied GGUF language model as a local service with llama.cpp and llama-server. Bind to 127.0.0.1 by default, expose the runtime's OpenAI-compatible chat endpoint, add a health check, cap concurrent requests, and read context size and GPU layer count from environment variables. Provide a Dockerfile, a curl smoke test, and a benchmark that reports time to first token, tokens per second, peak memory, and answer quality for the supplied Q4 and Q8 quantizations.