Octopusman388/flint-uncensored-GGUF

flint-uncensored — GGUF (Q4_K_M)

GGUF quantization of electroglyph/flint-uncensored, a Heretic-decensored version of RecursiveSI/flint (Recursive's 39.8B-A2.5B MoE coding model). All credit for the model goes to Recursive (original Flint) and electroglyph (decensoring). This repo only converts and quantizes; no further training. License: Apache 2.0, as upstream.

• File: flint-uncensored-Q4_K_M.gguf (22.5 GiB, 4.86 BPW), converted from the BF16 safetensors (sha256-verified), then quantized with llama-quantize.
• Requires llama.cpp with Flint support (upstream commit a37cc461, "models : add Flint model"; built from 4231b23d).
• The upstream converter needed two local patches for this checkpoint (transformers-5 export): skip model.rotary_emb.original_inv_freq, and map self_attn.sinks to the attention-sinks tensor. Without them conversion/loading fails ("blk.0.attn_sinks.weight not found").

Known issue: runaway reasoning ("looping") in agent harnesses

A user reported the model "kept looping" in the pi coding agent. What I found when replaying it:

• It is not a token-repetition loop and not a template/EOS problem (<|user|> and <|observation|> are already EOG tokens; tool calls are well formed; normal agentic tasks finish in seconds).
• On borderline/sensitive prompts the model can ramble in its reasoning for 10k+ characters with no answer ("No — ... OK here's what I remember... No —"). It is stochastic: 2 of 3 unbudgeted runs on one such prompt hit the token limit without ever answering.
• presence_penalty 1.5 did not help.
• Mitigation that worked: cap thinking, e.g. --reasoning-budget 4096 --reasoning-budget-message "I have thought enough. I will now answer directly.". All budgeted runs terminated (4/4 single-turn, plus two multi-turn tool-using agent runs of 8 and 12 turns with no repeated identical tool call). The budget caps the drift, it does not prevent it: one run still spent 14k chars (4 min) thinking before answering.
• I could not reproduce a true repeated-tool-call loop. If one occurs in your harness, it may come from harness behaviour I did not simulate. Only tested on one sensitive prompt plus simple coding tasks.

Suggested launch (RTX 4060 8 GB, 62 GB RAM)

llama-server -m flint-uncensored-Q4_K_M.gguf -c 131072 -ngl 99 -b 2048 -ub 1024 \
  --cache-type-k f16 --cache-type-v f16 --flash-attn on --jinja \
  --temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
  --n-cpu-moe 47 --threads 8 --load-mode none --parallel 1 \
  --reasoning-budget 4096 --reasoning-budget-message "I have thought enough. I will now answer directly."

Use -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 for the full native window (7.2 GB VRAM).

Measured decode speed (all experts on CPU), tokens/s:

depth	q8_0 KV	f16 KV
0	26-31	26
64k	17.6	21.7
131k	7.6	10.4

Prefill about 520 t/s at ub 1024. Numbers vary with CPU load (i7-12700H).

Safety

Abliterated: refuses 5/100 test prompts (vs 98/100 for the original). It can produce harmful or incorrect content; you are responsible for its use. Sampling per upstream: temp 1.0, top-p 0.95, top-k 20.