alimpfard/qwen3.8-27b-kv-approximation

Qwen3.8-27B-KVA: late-layer KV approximation

A 1.24B-parameter projector (r3; the r2 model was 931M) that approximates half of Qwen3.8-27B's prompt processing layers. As measured, this provides a 30-60% prefill performance improvement in exchange for ~4-5% perplexity (measurements below).

This repo contains only the projector (kva.* tensors, 0.5-1.3 GB depending on variant and quantization); which can be merged into any Qwen3.8-27B GGUF using the provided merge script.

A reference llama.cpp implementation that can read and use the kva tensors is provided on the kva branch in my fork of llama.cpp.

What the hell?

Qwen 3.8-27B has 64 layers, 48 gated-DeltaNet layers and 16 full-attention layers; during prefill the engine runs layers 0..31 as is, and uses this approximator to map the layer-31 residual of every token to the token-mixer inputs of each late layer:

late layer type	predicted per token	dims
gated deltanet (24 layers)	in_proj_qkv output (q, k, v before conv/silu), in_proj_a, in_proj_b	10240 + 48 + 48
full attention (8 layers)	k_proj, v_proj outputs (before k_norm / rope)	1024 + 1024

Afterwards, the engine runs some cheap state updates for the tokens (conv + silu, l2-norm, and deltanet scan, plus k_norm + rope + KV cache writes for the attention layers), and runs the last token of each ubatch through the late layers to produce exact logits on top of the approximated context.

The projector's own architecture is a simple RMSNorm -> Linear{5120 -> d} -> 3x[causal transformer block] -> RMSNorm -> one Linear head per late layer. The blocks attend only within the ubatch they are given (the engine's, 1024 tokens); nothing about the projector depends on prompt length.

	r2 (kva_r2_*)	r3 (kva_r3w_*)
width d	2560	3072
blocks	3	3
attention	20 heads @ 128, full causal	24 heads @ 128, causal within 1024-token blocks
SwiGLU FFN	7040	8448
trunk	254M	363M
mixer heads (32 layers)	677M	813M
drafter heads	none	residual layers 34, 48, 62 (kva.res.N) + the final hidden state (kva.final)
total	931M	1238M

Prefill does not need the drafter heads, but speculative decoders that read per-token hidden states (e.g. Qwen's MTP head, or DFlash2) hook into either the final hidden state, or residuals at specific layers; the drafter heads exist to provide them with real signals instead of fabricated nonsense.

This work was inspired by nowokay's blog post and the causal encoder-decoder split of DeepSeek-V4.1-Flash.

v1 Eval Results

Measurements are done on held-out text cut into windows of 1536 prompt tokens + 512 continuation tokens; the prompt is prefilled (exact vs. KVA) and the continuation is forced through the exact model on top of that cache. This report presents perplexity, and top-1% greedy matches per token. "wiki" is Wikitext-103, and "code" is a python code file.

llama.cpp engine, Q4_K_M base, (24 windows, q8_0 projector)

	exact	KVA	delta
wiki continuation ppl	5.949	6.214	+4.5%
code continuation ppl	2.109	2.207	+4.6%
greedy top-1 agreement with the exact model, wiki		91.3%
greedy top-1 agreement with the exact model, code		93.8%
prefill, ubatch 1024, RTX Pro 4500	1753 tok/s	2684 tok/s	1.53x
prefill, ubatch 1024, 2x Titan Xp (layer-split)	256 tok/s	404 tok/s	1.58x

Roughly, the quality loss is equivalent to going from Q4_K_M to unsloth's UD-Q3_K_XL. For scale, llama-perplexity reports ppl ratios of 1.007 (Q4_K_M), 1.003 (UD-IQ4_XS), 1.034 (UD-IQ3_XXS) and 1.055 (Q3_K_M) on wikitext (2k token chunks, 40 chunks) -- note that this is a different protocol from the report above.

Projector quantization and flash-attn (Q4_K_M base, 4500, 12 windows, -c 4096)

projector	file size	wiki ppl	code ppl	prefill ub 1024
q8_0	990 MB	6.0253	2.3087	2659 tok/s
q4_0	525 MB	6.0304	2.3113	2638 tok/s

From our measurements, q4_0 of the projector is more-or-less equivalent in this version; even though it has no meaningful impact on performance, it frees up ~400 MB of VRAM that can be used elsewhere.

Long prompts and the exact tail (round 3 projector, Q4_K_M base, 2 windows, wiki)

The projector's error does not grow with prompt length, but the model's continuation leans on the most recent few thousand tokens, and running those exactly recovers most of the loss; this does reduce the prefill performance as e.g. the last 4 ubatches of the prompt are run exactly. This mostly helps agentic work when incremental prompt updates have "important" low-signal data that cannot be approximated, e.g. exact file names.

prompt	exact	KVA, no exact tail	tail ~1k	tail ~2k	tail 4096
32k	5.29	6.27	5.63	5.49	5.33
96k	6.46	7.05	6.66	6.55	6.52

General failures with bad approximations show up as generation reproducing filenames or function calls incorrectly, then flailing about failing to find the now-misnamed file or function :^) Agentic work of ~128k ctx across a couple compactions has worked out fine with tail 2k in our experiments.

Other notes

• Quantized KV caches (-ctk q8_0 -ctv q8_0) work; the engine applies the same Hadamard rotation to the approximated K/V as the exact path does. Flash attention must be on (-fa on, the default on CUDA); the exact tail's V write does not support the transposed cache of -fa off.
• Tensor split (-sm tensor) and layer split across GPUs are supported.
• Abliterated / re-quantized variants of the base work unchanged; e.g. Huihui-Qwen3.8-27B-abliterated UD-Q4_K_XL: wiki 5.767 -> 6.008 (+4.2%), code 2.178 -> 2.302 (+5.7%), 12 windows.
• The engine prints kva: N of M prompt tokens approximated at warning level after each batch that used the approximated path, so you can tell from the log whether it engaged (LM Studio shows only warning-level library lines).

Training data

Round 2 as trained on 34M tokens, one sequence of at most 3072 tokens per step for 36k steps, sampled by sequence count from:

source	domain	sequences	tokens
Manusagents/GPT-5.5-…-Distillation-Dataset (coding + applied splits)	code, chat-templated	24.7k	16M
r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation (canonical split, with tools)	agentic, chat-templated	11.5k	12M
Salesforce/wikitext (wikitext-103-raw-v1, train)	prose, plain text	2.0k	6M
mondk/GptModel-CoT	chain-of-thought, chat-templated	47	0.07M

The CoT source contributed almost nothing in this round (47 sequences); the eval prompts (wikitext test, a held-out Python file) are not part of the training data.

Round 3 was trained on the same sources with more data, for 14k steps.

Limitations

• Quality was measured on English prose and Python (1536-token prompts; 32k / 96k / needle-at-60k for the long-prompt table). Other languages and domains are untested.
• Without the exact tail, long prompts lose fine detail (near-identical identifiers, tool-call formatting). Keep kva_tail_exact at the default or 2048.
• The projector attends over one ubatch (1024 tokens in training); ubatch sizes above that are untested.
• This projector only applies to the "base" Qwen3.8-27B architecture (quantized and abliterated variants are fine).
• Tested on CUDA only (Blackwell sm120 and Pascal sm61).