Quantization Formats — GGUF Deep Dive

Q4_K_M is the production sweet spot for 7B–13B models: 0.15 perplexity loss is imperceptible in human evaluation, 4× VRAM reduction enables larger context or smaller GPU. Use Q8_0 when output quality is critical (legal, medical, code generation). Never ship Q2_K in production — the quality degradation is visible in coherence and factual accuracy.