Learn
What is LLM quantization?
Last reviewed July 16, 2026
Quantization compresses a model by storing its weights at lower numeric precision: 8-bit or 4-bit integers instead of 16-bit floats. A 4-bit quant needs roughly a quarter of the original memory, which is what lets a 27B-parameter model run on a consumer GPU. The trade is quality: aggressive quantization measurably degrades output, mildly at 8-bit, more visibly below 4-bit.
Reading quant names
Local model builds carry tags like Q4_K_M or Q8_0: the number is bits per weight, the suffixes name the packing scheme. Lower bits mean a smaller download and lower RAM needs. Q4_K_M is the common sweet spot for chat use, while Q8_0 stays close to full quality at twice the size. The same model usually ships in several quants, so you pick per machine, not per model.
What actually gates local use
Memory, not compute: the whole quantized model must fit in GPU VRAM or system RAM, so the quant's size decides whether it runs at all, and your hardware decides how fast. A GPU raises speed; RAM decides fit. That is why hardware-fit checks work from your RAM and VRAM against each quant's stated requirements.
idapt's model directory has a hardware-fit filter: pair a computer and it shows which models and quants your machine can serve, with a recommended quant per model, installed through Ollama.
Frequently asked
See it in practice
The fastest way to understand it is to use it. Start free, no credit card.
Try idapt free