mudler/Clover-35B-A3B-Preview-APEX-GGUF

⚡ Each donation = another big MoE quantized

I host 30+ free APEX MoE quantizations as independent research. My only local hardware is an NVIDIA DGX Spark (122 GB unified memory), enough for ~30-50B-class MoEs, but bigger ones (200B+) require rented compute on H100/H200/Blackwell, typically $20-100 per quant.
If APEX quants are useful to you, your support directly funds those bigger runs.

🎉 Patreon (Monthly)  |  ☕ Buy Me a Coffee  |  ⭐ GitHub Sponsors

Clover-35B-A3B-Preview APEX GGUF

APEX (Adaptive Precision for EXpert Models) quantizations of Diffuspace/Clover-35B-A3B-Preview.

Brought to you by the LocalAI team | APEX Project

A block-diffusion hybrid MoE model, quantized to nine tiers.

Status: preview. Results below are a first pass with small sample sizes.

Benchmark Results

Measured with a llama.cpp build of PR 31306 (head 4838ffb) on an NVIDIA GB10. Perplexity and KL divergence are on wikitext-2 (580 chunks), KL divergence against BF16 reference logits. HellaSwag, Winogrande, ARC, MMLU and TruthfulQA use the llama.cpp multiple-choice mode with 400 tasks each (about ±2 to 2.5 points of noise, so the top six tiers are not separable on these). Speed is pp512 / tg128 in tokens per second.

Tier	Size	PPL ↓	KL mean ↓	Same top-p % ↑	HellaSwag	Winogrande	ARC	MMLU	TruthfulQA	pp512	tg128
BF16 (reference)	69 GB	11.478	—	—	—	—	—	—	—	—	—
Quality	22.8 GB	11.536	0.0853	86.93	71.75	66.00	52.84	40.96	36.60	1504	17.97
I-Quality	22.8 GB	11.460	0.0752	87.56	70.50	68.25	51.51	40.44	35.74	1504	18.10
Balanced	25.3 GB	11.360	0.0826	87.20	72.50	67.50	52.17	40.50	37.21	1485	17.75
I-Balanced	25.3 GB	11.283	0.0740	87.97	71.75	66.25	50.50	40.05	37.09	1478	17.91
Compact	16.5 GB	11.603	0.1890	80.39	70.25	68.00	51.51	39.53	35.37	1419	21.02
I-Compact	16.5 GB	11.674	0.1583	82.07	69.25	66.00	52.51	39.60	35.25	1559	20.84
I-Mini	13.5 GB	13.876	0.3114	75.17	68.50	64.50	49.16	38.76	34.88	1441	22.07
I-Nano	11.5 GB	15.729	0.3990	71.98	66.50	62.75	45.15	37.66	32.31	1620	22.20
I-Micro	10.6 GB	16.854	0.4593	70.06	65.00	58.50	41.47	36.56	31.46	922	22.37

Notes:

• The imatrix files (I-) lower KL divergence at every size where both exist.
• Balanced and I-Balanced show a lower perplexity than BF16 on wikitext-2. Treat that as a quirk of the test text, not as a quality gain.
• KL divergence is higher than on dense-attention MoE models of this size, even for the largest tiers. The BF16 reference itself has a high perplexity on this text, and this is a diffusion hybrid, so compare tiers with each other rather than with other models.
• Token generation gains little speed below 16 GB; I-Micro prompt processing is slower than the other tiers.

Files

File	Size	For
Clover-35B-A3B-Preview-APEX-Quality.gguf	22.82 GB	highest quality
Clover-35B-A3B-Preview-APEX-I-Quality.gguf	22.82 GB	highest quality, imatrix
Clover-35B-A3B-Preview-APEX-Balanced.gguf	25.27 GB	general purpose
Clover-35B-A3B-Preview-APEX-I-Balanced.gguf	25.27 GB	general purpose, imatrix
Clover-35B-A3B-Preview-APEX-Compact.gguf	16.54 GB	consumer GPUs
Clover-35B-A3B-Preview-APEX-I-Compact.gguf	16.54 GB	consumer GPUs, imatrix
Clover-35B-A3B-Preview-APEX-I-Mini.gguf	13.47 GB	small, imatrix only
Clover-35B-A3B-Preview-APEX-I-Nano.gguf	11.46 GB	smaller, imatrix only
Clover-35B-A3B-Preview-APEX-I-Micro.gguf	10.58 GB	smallest, imatrix only

I- files use an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces, Wikipedia). Quality, Balanced and Compact also ship without it.

What is APEX?

APEX is a quantization strategy for Mixture-of-Experts models. It classifies each tensor by role (routed expert, shared expert, attention) and applies a layer-wise precision gradient: the first and last layers keep higher precision, middle layers compress harder, and the always-active shared expert stays high.

The reason: in an MoE model the routed experts are most of the weights, but only a few of them fire for any given token. They tolerate lower precision than the parts every token passes through. See the APEX project for the recipes and scripts.

The model

Clover-35B-A3B-Preview is a block-diffusion hybrid Mixture-of-Experts model (clover_moe) with 34.7 B parameters and about 3 B active per token. Its tensor layout matches the Qwen3.5 MoE family (256 routed experts, hybrid linear/full attention). The small diffusion tensors stay in F32.

Usage

llama.cpp master does not load these files yet. Support for the clover_moe architecture is the open pull request ggml-org/llama.cpp#31306. Build llama.cpp from that PR (head 4838ffb) or from a later merge of it.

llama-cli -m Clover-35B-A3B-Preview-APEX-I-Mini.gguf -p "Your prompt" -ngl 99

Once the architecture is merged into llama.cpp, it will also run with LocalAI. Until then LocalAI cannot load these files.

License

These files are derivatives of Clover-35B-A3B-Preview by Diffuspace and fall under the base model license (Apache-2.0). Keep the license and attribution when you redistribute them.

Credits

APEX is brought to you by the LocalAI team. Built on llama.cpp.