Benchmarking & Performance Tuning

Always offload all layers to GPU when VRAM allows — each CPU layer adds ~8ms latency penalty per generated token. Context length is the biggest VRAM consumer after model weights: 32K context on a 7B model costs ~16 MB of KV cache. Batch size >1 only helps throughput when handling concurrent requests; for single-user interactive use, keep it at 1 for lowest TTFT.