vcruz305/Ornith-1.5-35B-A3B-GGUF

Ornith-1.5-35B-A3B — GGUF (1-3 bit IQ ladder)

GGUF quantizations of ornith-ai/Ornith-1.5-35B-A3B, a sparse MoE model (256 experts, 8 active per token) with hybrid linear/full attention and an MTP (multi-token-prediction / NextN) head. This repo covers the low-bit IQ ladder from IQ1_S through IQ3_M, quantized with a mixed-corpus importance matrix for better accuracy at the smallest sizes.

Imatrix

Built with a mixed calibration corpus rather than plain wikitext, to better match this model's actual use cases and its MoE expert-coverage needs:

• ~45% agentic tool-use data (chat-template-rendered)
• ~35% code data (HumanEval excluded, to avoid contaminating any code-eval benchmarks run against these quants)
• ~20% wikitext-103 (general language coverage)

~11M characters, tokenized with the model's own chat template, run through llama-imatrix (-c 1024 -b 1024 -ub 1024 -np 2). Final calibration PPL: 3.9309 +/- 0.00885.

The blk.40 (MTP/NextN) pin

blk.40 is the model's MTP/NextN head. It sits structurally outside the normal forward pass, so no calibration corpus — however good — can ever give it real importance-matrix coverage (llama-imatrix never activates it during a forward pass, confirmed in the load logs of every serving attempt). Quantizing it at the same bit-width as the rest of the ladder fails outright at IQ1_S with "Missing importance matrix for tensor blk.40.attn_k.weight in a very low-bit quantization".

Every quant in this repo pins blk.40.* to Q8_0 via --tensor-type blk.40=q8_0 regardless of the overall quant level, and this is verified programmatically after each quantize (every blk.40.* tensor confirmed Q8_0 or F32 before upload). This adds a small, fixed amount of size to every file but keeps the MTP head numerically sound at every bit-width in the ladder.

Files

Quant	Size	Notes
IQ1_S	8.38 GB	Smallest, most aggressive. Expect noticeable quality loss vs FP.
IQ1_M	9.14 GB
IQ2_XXS	10.4 GB
IQ2_XS	11.41 GB
IQ2_S	11.55 GB
IQ2_M	12.56 GB
IQ3_XXS	14.52 GB
IQ3_XS	pending	uploading as part of the same ladder run
IQ3_S	pending
IQ3_M	pending	Best quality in this ladder, closest to the source BF16.

(Sizes for IQ3_XS/IQ3_S/IQ3_M will be filled in as each finishes quantizing and uploading.)

Usage

llama-server -m Ornith-1.5-35B-IQ2_M.gguf -ngl 99 -c 8192

Standard llama.cpp builds will load and run these normally; the MTP/NextN tensors are inert unless served with an MTP-aware fork (--spec-type draft-mtp), in which case they can be used for self-speculative decoding.

Source

• Base model: ornith-ai/Ornith-1.5-35B-A3B
• Quantized with a custom CUDA build of llama.cpp targeting GB10 (Grace Blackwell, sm_121a)