llama.cpp Internals

KV cache is why context length drives VRAM requirements — every cached token costs memory (roughly 0.5 MB/K tokens for a 7B model). Speculative decoding gives 2–3× speed gains when the draft model (e.g., Phi-3.5-mini) agrees with the large model >70% of the time. Grammar constraints add ~15–30% latency but guarantee 100% schema compliance.