0xBakeer/TandemLLM-Flint-EXL3-4.0

Flint in EXL3, 4.0 bits per weight

These are EXL3 weights of Recursive's Flint at 4.0 bits per weight, 20.6 GB in all. We converted them from the BF16 checkpoint, RecursiveSI/flint at revision 2a247be0, with exllamav3 1.6.2 and the patch in convert/, which adds the Flint architecture. Every projection is at 4 bits: attention, the two dense layers, the shared experts and the routed experts. The output head is at 5 bits. The router, the norms and the attention sinks stay at 16 bits.

This is the fast size. One decoded token reads 1.34 GB of these weights, two thirds of what the 6.0 pack reads, and it costs quality: the table below shows how much. Take the 6.0 pack when the text matters more than the speed.

We serve these weights with TandemLLM, our inference engine for one DGX Spark. Flint has 49 layers. Two are dense; the other 47 have 256 routed experts, of which 6 run a token, plus one shared expert. Attention has 32 query heads and 4 KV heads with a learned sink per head. 35 layers use a 513-token sliding window with rotary, and 14 full layers use no positional encoding. The vocabulary has 150,144 tokens and the context is 262,144 tokens.

One thing loaders need to know

The routed experts' SwiGLU activations get large in the top layers: |silu(gate) x up| reached 260,000 at layer 48 on a 2,048-token text, past fp16's 65,504. The patch keeps those activations in range: it stores every routed expert's up projection divided by 128 and multiplies the routing weights by 128. config.json says so: quantization_config.expert_interm_div is 128. A loader that ignores it gets expert outputs 128 times too small.

Quality against BF16

We compared the pack with the BF16 checkpoint on held-out text the model cannot have seen: TandemLLM's own docs and Python, written after Flint's training cutoff, plus wikitext-2. KL is the mean KL divergence from BF16 a position, and top-1 is how often both pick the same next token.

run	text	KL	top-1	perplexity	BF16 perplexity
exllamav3 eval/model_diff.py	docs, 50 × 2,048 tokens	0.194	79.7 %	32.37	30.80
exllamav3 eval/model_diff.py	Python, 50 × 2,048	0.252	85.1 %	9.726	9.599
exllamav3 eval/model_diff.py	wikitext-2, 50 × 2,048	0.263	82.9 %	23.32	23.07
TandemLLM engine	docs, 8 × 1,024	0.160	80.5 %	31.65	29.88

Part of that gap is the model itself. On those 8 rows, two BF16 forwards of the same weights already differ by KL 0.034 and agree on 92.9 % of top-1. Flint's router picks 6 of 256 experts by sigmoid scores that sit close together, so one BF16 rounding of the residual stream changes the expert choice for 2 % of tokens at layer 2 and for 46 % by layer 46. The 6.0 pack lands at KL 0.070 on the same docs, so going from 6 to 4 bits costs about 0.12 on top of that. The model_diff runs are in quality/.

Speed on a DGX Spark

One DGX Spark (GB10, 128 GB), nothing else running. TandemLLM 0.14.0. One request at a time, 512 greedy tokens, thinking off, a fresh server for each row. The game file row asks for a whole HTML game and runs 2,048 tokens. Edit and copy are asks whose answer repeats much of the prompt. Each cell is one run, in tok/s.

tok/s	prose	code	chat	edit	copy	game file
plain	136.6	133.2	133.3	128.2	128.2	129.3
lookup + StairCut (default)	135.4	139.8	134.0	142.3	480.0	149.5

Speed is greedy with thinking off. A fresh TandemLLM install thinks at effort medium; docs/setup.md says how to change it and which effort each published number used. In every drafted run the text was the same as plain greedy, token for token. Flint has no draft layer, so TandemLLM drafts from text already in the prompt. StairCut sets the block size from the pack's price file, the measured cost of a verify step by block size and context length. After a round with no draft, the engine takes 1, 2, 4, then 8 plain steps before it looks again.

Long prompts slow decoding, because every token reads the 14 full layers' keys and values. Measured in the engine, 64 greedy tokens at each point:

context	4k	16k	64k	128k	256k
plain decode, tok/s	123.0	101.7	61.0	39.5	23.0

The lookup gains more there. Behind a 124,000-token prompt of code, a copy ask ran at 79.5 tok/s against 41.3 plain and an edit ask at 129.7 against 42.4, with the same text. A 4,096-token prompt reads at 5,375 tok/s. The weights take 19.3 GiB in memory and load in about 9 s; at the full 262,144-token window the engine peaked at 27.6 GiB.

Long-context cache (FP8), from TandemLLM 0.14.1 on

TandemLLM can keep the 14 full layers' keys and values as FP8 (e4m3) codes with one fp32 scale per head and token: 14,784 bytes a token instead of 28,672. It is off by default, because the output is then no longer the exact model's.

We measured it on this pack against the exact cache. The prompt was one long source file, and we scored the last 1,024 tokens:

prompt	KL	top-1 agreement	perplexity, exact / FP8
32,768 tokens	0.070	93.8 %	7.165 / 7.465
122,880 tokens	0.050	95.7 %	4.053 / 4.132

Plain decode in the engine, 64 greedy tokens at each point, one run each:

context	64k	128k	256k
exact cache, tok/s	60.6	39.3	23.05
FP8 cache, tok/s	79.4	57.1	36.6

Reading a prompt gets 5 to 12 % slower, since each chunk decodes the cached rows back to BF16.

To turn it on in the Setup Guide (the dashboard's #/setup), pick Flint, go to the Tools step and switch on "Long-context cache (FP8)" under Features. Then finish Review and Install. To turn it on from the command line, add this line at the end of ~/TandemLLM/run/ops/serve.env:

QWEN38_FLINT_KV_FP8=1

Then run ~/TandemLLM/bin/tandem restart. curl -s localhost:8000/health shows "kv_cache": "e4m3" under weights when it is on. To turn it off, switch it off in the Setup Guide, or delete the line and restart.

Tool calls: tool-eval-bench, from TandemLLM 0.14.1 on

TandemLLM 0.14.1 on one DGX Spark, 2026-10-22. tool-eval-bench 2.8.1.dev13 (commit f2e368b), seed 42, temperature 0, one request at a time, one run a row (n=1), a restart before each row. The server ran the Setup Guide's defaults for this pack: one slot, the shipped effort medium, the conductor on, the FP8 cache off. The raw rows send their own effort, thinking off or xhigh. The tools rows think at the server's medium, and the bench-pk rows at the level of each turn's class. Score, then the time the row took:

row	full suite, 69 scenarios	hard mode, 28 scenarios
raw, thinking off	75 (2 min)	36 (1.8 min)
raw, thinking xhigh	85 (4.7 min)	71 (5.5 min)
conductor, bench-pk (effort by turn class)	81 (5.1 min)	73 (3.5 min)
conductor, tools (effort medium)	83 (4.1 min)	80 (2.2 min)

Raw rows send X-Tandem: raw, so the model answers alone; thinking off is reasoning_effort: none. The conductor rows pin a profile with X-Tandem-Profile: tools is what the conductor picks for a client that offers tools, and bench-pk is a test profile TandemLLM does not ship. Hard mode at n=1 varies a lot between runs, so we claim no ranking between the packs. The measurement file has every row with its misses.

How to run it

TandemLLM runs this pack from release 0.14.0 on. On a DGX Spark with TandemLLM 0.14.0 installed:

~/TandemLLM/run/ops/use-profile.sh flint-exl3-4.0

use-profile.sh downloads this repository at a pinned commit, checks each file against SHA256SUMS and starts the engine on port 8000 with the lookup drafting. Flint decides on each turn whether to reason in a <think> block; TandemLLM's effort levels cap that block, and thinking off answers at once.

These are EXL3 files with one architecture stock exllamav3 does not have: a loader needs the patch in convert/ (or its own Flint code) and must read expert_interm_div. We have not tested other EXL3 loaders.

How we made it

convert/exllamav3-1.6.2-flint.patch applies to exllamav3 v1.6.2 (commit 4c84273). It adds exllamav3/architecture/flint.py, a guard in the quantizer for all-zero weights, and an option in eval/model_diff.py to evaluate on a local text file. convert/README.md has the commands. We used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook, on one RTX PRO 6000. Before the conversion we checked the converter's forward against a plain fp32 implementation of the published model description: a tiny random Flint per layer within fp16 noise, and all 49 real layers at cosine 0.99998 or better. quality/ holds those checks and the version file.

License

Flint is released under Apache-2.0, and so are these weights. LICENSE is Recursive's, and NOTICE says where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed. Everything else about the model, including its intended use and its benchmarks, is in Recursive's model card.

Credits

Recursive made Flint and its BF16 checkpoint. turboderp made exllamav3 and the EXL3 format. TandemLLM is ours.