anik-jha/Flint-40B-A2.5B-8GB-GGUF

Recursive Flint 40B-A2.5B for coding on an 8 GB GPU (expert-pruned GGUF)

A coding-focused build: experts were chosen on calibration data that was mostly code (70% of the mix is code, coding-agent trajectories and function calls, and two thirds of Flint's own reasoning replies answer coding prompts), and only coding was benchmarked.

Run Flint fully on an 8 GB card. unsloth's smallest Flint GGUF (UD-IQ1_M) is 10.6 GB, more than an 8 GB GPU holds before any context. These GGUFs remove half of the routed experts in every layer, so the whole model and a 32K context fit on an 8 GB or a 12 GB GPU. They are cut from unsloth's 2-bit GGUF (UD-IQ2_XXS, for the 8 GB build) and 3-bit GGUF (UD-Q3_K_XL, for the 12 GB build), and nothing is re-quantized: every tensor that remains is byte for byte unsloth's.

	Flint Q8_0 (reference)	This repo: 8 GB	This repo: 12 GB
File	39.4 GB	6.1 GB	9.5 GB
GPU memory at 32K context, 8-bit KV cache (measured at server start)	39.9 GB	6.8 GB	10.0 GB
GPU memory at 32K context, 16-bit KV cache (measured at server start)	40.3 GB	7.2 GB	10.5 GB
HumanEval+ (pass@1, reasoning on)	0.927	0.799 (86.2% kept)	0.811 (87.5% kept)
MBPP+ (pass@1, reasoning on)	0.804	0.680 (84.5% kept)	0.704 (87.5% kept)
Replies that hit the 32K-token limit (HumanEval+ / MBPP+)	0 / 0	4 / 16	3 / 2

Sizes are in GB the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card. The Hugging Face file list counts in decimal, so it shows the 8 GB build's file as 6.55 GB.

The 8 GB build keeps about 85% of the reference model's coding scores (95% CIs: 79.9 to 92.4% on HumanEval+, 79.7 to 89.3% on MBPP+) at one sixth of its size. The reference is unsloth's 8-bit Q8_0, so these figures combine the cost of pruning with the cost of 2-bit or 3-bit quantization; the unpruned 2-bit model was not evaluated.

Files

File	Use
Flint-40B-A2.5B-8GB-REAP50-UD-IQ2_XXS.gguf	8 GB GPUs. Pruned from unsloth's 2-bit UD-IQ2_XXS
Flint-40B-A2.5B-12GB-REAP50-UD-Q3_K_XL.gguf	12 GB GPUs. Pruned from unsloth's 3-bit UD-Q3_K_XL; slightly higher scores, fewer runaway replies
selections/flint-reap50.json	The exact experts kept in every layer
eval/	Scores with paired CIs, per-problem results, and provenance (source revisions and hashes, llama.cpp commit, calibration, expert-score stability, tensor verification)

40B-A2.5B names the parent model. REAP50 means 50% of each layer's routed experts were kept.

Run it

Use a llama.cpp build that includes Flint support (ggml-org/llama.cpp PR #30928) and its chat-template fix (#30929). Evaluated with llama.cpp at commit 36cbf415, plus a small local patch that returns malformed chat output as plain text instead of an HTTP error; it never triggered in any run behind this card.

llama-server -m Flint-40B-A2.5B-8GB-REAP50-UD-IQ2_XXS.gguf -ngl 99 -c 32768 \
  -ctk q8_0 -ctv q8_0 -fa on --temp 1.0 --top-p 0.95 --top-k 20

• Flint always reasons: its chat template has no thinking switch. The sampling settings are Recursive's recommended ones, and the ones the benchmarks used.
• Set a token limit on replies. The pruned builds sometimes keep reasoning without finishing (see the table), and they reason longer than the full model: the 8 GB build's median HumanEval+ reply was 3,596 tokens against 1,098 for the reference.
• Memory was measured with these settings: one slot, every layer on the GPU, 8-bit KV cache, flash attention on. With a 16-bit KV cache (drop -ctk, -ctv) the 8 GB build allocates 7.2 GB at 32K context.
• LM Studio and Ollama have not been tested.

How it was made

Flint routes each token to 6 of 256 experts in each of its 47 MoE layers, plus one shared expert; the routed experts are 95% of its parameters. We measured how much each expert contributes on 5.8 million tokens: a 2,047-conversation calibration mix (code instructions 35%, general chat 30%, software-engineering agent trajectories 17.5%, function calling 17.5%; 2.95M tokens) and 768 of Flint's own reasoning replies to prompts from that mix (512 coding, 256 general chat; 2.82M tokens). Experts were scored with REAP (router weight times expert output size, averaged over the tokens routed to the expert), and the top 128 of 256 were kept in every layer. Every layer keeps the same number because llama.cpp's Flint loader reads one expert count for the whole model. The shared expert, attention, and the two dense layers are untouched.

The pruning is a byte copy: kept experts, their router rows, and every other tensor are identical to unsloth's file, checked tensor by tensor (779 tensors) for both builds. Expert scores were stable: rankings from the two halves of the calibration set correlate at a median of 0.986 across layers (0.967 in the least stable layer).

Method background: Half the Experts, All the Code, accepted at the COLM 2026 Workshop on Efficient Reasoning.

Evaluation details

• Code: EvalPlus HumanEval+ (164 problems) and MBPP+ (378), served by llama-server, reasoning on, temperature 1.0, top-p 0.95, top-k 20, fixed seed, one sample per problem, up to 32,768 generated tokens. The answer after the reasoning block was sanitized and scored with EvalPlus 0.3.1. A reply that hit the token limit counts as a failure.
• Against the reference, the 8 GB build's drop is -12.8 points on HumanEval+ (95% CI -18.9 to -6.7) and -12.4 on MBPP+ (-16.4 to -8.5); the 12 GB build's is -11.6 (-17.7 to -6.1) and -10.1 (-13.8 to -6.6). All four are significant (McNemar p < 0.001). Paired bootstrap CIs and per-problem results are in eval/.
• One sample at temperature 1.0 per problem: the CIs cover the choice of problems, not run-to-run sampling noise.
• Memory: GPU memory of the llama-server process (nvidia-smi), model fully offloaded, one slot, read once the server was ready and before any request, measured on an NVIDIA DGX Spark (unified memory). llama.cpp's own breakdown for the 8 GB build at 32K context with the 8-bit KV cache: 6,114 MB weights + 522 MB KV + 129 MB compute. Memory on an actual 8 or 12 GB discrete GPU was not measured; on a GPU that also drives a display, leave some headroom.

Limitations

• Flint is built for coding agents. Only single-function coding (HumanEval+, MBPP+) was measured; agentic coding, tool calls, long-context use, general knowledge and other languages were not, and may degrade more.
• The pruned builds reason longer than the full model, so answers take longer to arrive than the token rate alone suggests.

License and attribution

Unofficial community derivative of Recursive Flint, pruned from unsloth's GGUF quantizations; not affiliated with or endorsed by Recursive or Unsloth. Distributed under the Apache License 2.0, the license of the original model; the full text is in LICENSE. The GGUF metadata fields general.name ("Flint Preview") and general.license.name ("flint-preview") are carried over unchanged from unsloth's files; both source repositories declare Apache 2.0.