HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT

Qwen3.6-35B-A3B Q4_K_M — CPU-optimized (Intel Xeon Platinum 8559C)

One model, one quant, one machine: Qwen3.6-35B-A3B-UD-Q4_K_M.gguf (unmodified file from unsloth/Qwen3.6-35B-A3B-GGUF) plus a llama.cpp patch (base commit de5122d) tuned for this CPU. The GGUF is not changed; all gains come from code. Q4_K_M is a mixed quant: besides Q4_K it contains Q5_K (37 MoE down projections), Q6_K (lm_head, 3 down projections) and Q8_0 (attention/DeltaNet/shared expert), so the patch touches the kernels for those tensor types too.

Results (16 cores, taskset -c 0-15 -t 16 -fa on --load-mode mlock, 4 interleaved rounds)

variant	pp256 t/s	tg128 t/s	pp2048 t/s
stock llama.cpp (ub 512)	170.3	24.9	177.8
stock llama.cpp (ub 1024)	-	-	161.3
this patch (pp2048 with -b 1024 -ub 1024)	273.9	39.4	282.1 (median 301.2)

Speedup (means): pp256 1.61x, tg128 1.58x, pp2048 1.59x vs stock default (1.75x vs stock at ub 1024). Raw data: benchmarks/q4km.jsonl, benchmarks/summary.md. The host was shared (load average 50-80), so single runs vary by roughly +-10-15% or more (one pp2048 round came in at 211); stock and patched runs are interleaved to cancel drift. For long prompts use -b 1024 -ub 1024 (about +10% prefill with this patch).

Accuracy

Not bit-exact (integer dot products are exact, but float accumulation order and batch-path selection differ). Measured against stock Q4_K_M logits on 32x512 tokens (benchmarks/kld_*.log):

compared with stock default	mean KLD	same top-1
stock --no-repack (another valid stock kernel path)	0.0045 +- 0.0007	99.63%
this patch	0.0036 +- 0.0005	99.74%

PPL(patch)/PPL(stock) = 1.0005 +- 0.0012. The deviation is the same size as switching between two stock kernel paths, and not statistically distinguishable at this sample size. Greedy decoding of a short prompt is byte-identical to stock. The test text is repetitive (PPL ~1.05), so it is a weak discriminator of absolute quality.

What the patch does (patches/qwen36-q4km-cpu-opt.patch)

1. AVX512-VNNI vec_dot for Q4_K and Q6_K, and a new one for Q5_K, with L2 software prefetch (GGML_K4_PF=40, GGML_Q6_PF=44, GGML_K5_PF=48 blocks ahead).
2. Q4_K 8x8 AVX2 repack off by default (GGML_REPACK_Q4K=1 re-enables). This also lets the upstream tiled VNNI K-quant path handle MoE prefill.
3. output.weight (Q6_K lm_head) is kept out of the AMX buffer, so decode uses the VNNI vec_dot (2.5 ms vs 4.6 ms per token; GGML_AMX_Q6K=1 restores the AMX path). About +9% decode.
4. AMX M=1 kernel uses 32-wide tasks when N <= 1024 (the 512-row shared-expert matvecs used only 8 of 16 threads). About +5%.
5. Upstream tiled MoE prefill: expert row groups 64 -> 32 so all 16 threads work on the 512-row gate/up matrices (only 8 of 16 were busy). About +15% prefill.
6. GATED_DELTA_NET: fused 2-pass AVX512 update for the scalar-gate, S_v=128 case (the reference does 4 passes with per-row helper calls). 2x faster op in prefill (18 -> 9 ms per layer-call at 1k tokens); only the dot rounding order changes.
7. Recurrent-state fast path (single sequence, qwen35moe only): new op ggml_gated_delta_net_cache reads the Gated-DeltaNet state straight from the cache (ids input) and updates it in place, so the per-layer GET_ROWS gather (2 MB), the state memcpy and the CPY back disappear (CPU backend only; falls back to the old graph for multi-sequence, rollback or non-fused cases; disable with LLAMA_GDN_NO_INPLACE=1). Numerically identical, about +3% decode (the traffic moves into the op, net saving ~0.8 ms/token).
8. Dense Q8_0 prefill (AMX): new register-accumulating kernel. The generic AMX kernel read-modify-wrote the float C tile through memory and converted the A scale per row for every 32-wide K block (about 4-5 TFLOPS achieved); the new one keeps a 16x16 output tile's 16 row accumulators in zmm registers across all K blocks, pre-converts the A scales, and overlaps the next block's AMX product with the AVX epilogue. Same per-block math in the same order, so results are bit-identical (KLD unchanged). About +11% prefill overall (median 266 -> 296 t/s at pp512). GGML_AMX_Q8_REGACC=0 restores the old kernel.
9. GET_ROWS of the Gated-DeltaNet recurrent state (524288 floats, 30x/token) is split across threads. About +22% decode on its own.

optional-profiler.patch adds GGML_PROF=1 per-op timing (apply separately).

Build / run

./build.sh
taskset -c 0-15 llama.cpp/build/bin/llama-cli -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -t 16 -fa on --load-mode mlock

Notes / negative results

• --load-mode mlock mainly removes slow outliers (default mmap sometimes dropped to 18-26 t/s on the shared host); the best-case speed is the same. Needs ~22 GB lockable memory.
• No measurable gain (A/B with correct library paths): contiguous per-thread row split for matvec, --poll 0, --no-mmap, THP, a 512-bit Q8_0 vec_dot instead of the AMX M=1 kernel (equal at best), MTP speculative decoding (~+6%, benchmarks/mtp-optional.txt).
• The new Q5_K kernel is verified against the reference implementation (2000 random blocks, max rel. err 5.6e-5); its end-to-end gain is small (about +1-2%, within noise).
• MoE prefill already runs at roughly 7 TFLOPS-equivalent (about 70% of the AVX512-VNNI peak), so an AMX MoE path was judged not worth it. Swapping the AMX task order (weight-block major) for better weight reuse was tried and was 8% slower.
• Validated only on Xeon 8559C with 16 cores; other CPUs are not tested.