Choosing Quantized Code LLMs by File Size and Energy per Solved Task
Abstract
A developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A common rule says a larger model at lower precision tends to beat a smaller one at higher precision. We ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2 and priced each in energy per solved task on 542 EvalPlus tasks. The result is a selection guide indexed by Ollama's listed file size, with an evidence label on each pick. At 1 and 2 GB it picks the q4_K_M files of the 1.5B and 3B models, both on the correctness and energy frontier under every accounting. At 3 GB as listed, the larger-model rule picks 7B-q2_K, the only measured 7B file that fits, and there it does not pay. That file brings no significant gain in solves over the 2 GB pick, 3B-q4_K_M, and the 90% CI limits any 3B-q4_K_M deficit to 2.4 points. Yet 7B-q2_K needs 2.03 times the energy per solved task on CPU, GPU and memory rails, or 1.66 to 1.81 times on whole-system power. On HumanEval+, five alternating session pairs give a ratio of 1.925 against 1.956 in the main grid. At 8 GB, 7B-q6_K buys a significant gain over the 2 GB pick at a higher energy price. From 1.5B up, q4_K_M trails the highest precision measured at its size by less than 3 points (90% CI). We release per-task results, power logs and the analysis script.
Community
I measured 24 Ollama GGUF files of Qwen2.5-Coder on a fanless MacBook Air M2 over 542 EvalPlus tasks and priced each file in energy per solved task. At 3 GB as listed, 7B-q2_K brought no significant gain in solves over 3B-q4_K_M and needed 2.03 times the energy per solved task on CPU, GPU and memory rails (1.66 to 1.81 times on whole-system power). Per-task results, power logs and the analysis script are at https://doi.org/10.5281/zenodo.24016482
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Measuring the Energy Cost of Completing Tasks with Quantized Language-Model Agents (2026)
- Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms (2026)
- Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning (2026)
- Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs (2026)
- Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter (2026)
- FlagRot: Selective Rotation for Low-Bit KV Caches (2026)
- metalSPARSE: A 2:4 Semi-Structured Sparse GEMV Kernel for Apple Silicon GPUs (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper