Quantization stores weights more compactly
Quantization changes how the model's numerical weights are represented. Instead of keeping every value at a higher precision, a quantized format uses fewer bits and a scheme for reconstructing useful values during inference.
Fewer bits usually means less weight memory. It does not mean the model has fewer parameters: it is a more compact representation of the same learned structure.
Bits per weight is a useful planning signal
Bits-per-weight gives a rough way to compare memory pressure. A 4-bit candidate generally needs less weight storage than an 8-bit candidate for the same model, although metadata, packing, and runtime buffers mean the final file is not a perfect parameter-count multiplication.
Names such as Q4_K_M, Q5_K_M, and Q8_0 identify particular quantization schemes. They should be treated as candidate formats, not as a universal quality ranking across every model family.
The practical trade-off
Higher-precision candidates usually preserve more numerical detail but require more memory. Lower-precision candidates can make a model fit on consumer hardware, but aggressive quantization can change outputs and may be a poor choice for a particular workload.
LLMGauge recommends among the candidates recorded for a model. Its recommendation is based on fit and the supplied bits-per-weight metadata; it is not a benchmark, blind quality test, or guarantee that one conversion is best for every task.