Instructions to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF
# Run inference directly in the terminal:
llama cli -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF
# Run inference directly in the terminal:
llama cli -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Use Docker

docker model run hf.co/Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Llama new
LM Studio
Jan

vLLM

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Ollama
How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with Ollama:
```
ollama run hf.co/Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF
```
Unsloth Desktop

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with Docker Model Runner:
```
docker model run hf.co/Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF
```

Lemonade

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Run and chat with the model

lemonade run user.ThinkingCap-Qwen3.8-27B-MixC-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

ThinkingCap-Qwen3.8-27B - MixC GGUF (KLD-guided mixed quant, 15.1 GiB)

A single mixed-precision GGUF of bottlecapai/ThinkingCap-Qwen3.8-27B, built from a per-tensor-group sensitivity study instead of a stock preset.

Result: at 15.09 GiB, MixC has a lower KL divergence than the official Q4_K_M (16.25 GiB) on code and on held-out reasoning traces, and is on par on Wikipedia text. It fits a 262k context with 2 slots on a 24 GB GPU and runs at the same raw speed as Q4_K_S.

File	Size	Notes
`ThinkingCap-Qwen3.8-27B-MixC.gguf`	15.09 GiB	text model + embedded MTP head (blk.64)
`ThinkingCap-Qwen3.8-27B-reasoning.imatrix.gguf`	13 MB	importance matrix used here (reusable for other quants)

Vision is not included. The official projector mmproj-ThinkingCap-Qwen3.8-27B-f16.gguf should work with it, but I did not test it.

Recipe

Source: the official f16 GGUF (bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF). Base preset: Q4_K_S. Then the following overrides are applied:

llama-quantize --imatrix ThinkingCap-Qwen3.8-27B-reasoning.imatrix.gguf \
  --tensor-type ffn_gate=iq4_xs --tensor-type ffn_up=iq4_xs \
  --tensor-type ssm_alpha=q8_0 --tensor-type ssm_beta=q8_0 \
  --tensor-type "attn_k\.weight=q8_0" --tensor-type "attn_v\.weight=q8_0" \
  --tensor-type ffn_down=q5_K \
  ThinkingCap-Qwen3.8-27B-f16.gguf ThinkingCap-Qwen3.8-27B-MixC.gguf Q4_K_S

Tensor group	Q4_K_S	MixC
ffn_down (64)	Q4_K, Q5_K on 8 layers	Q5_K
ffn_gate, ffn_up (64)	Q4_K	IQ4_XS
ssm_alpha, ssm_beta (48 linear-attention layers)	Q4_K	Q8_0
attn_k, attn_v (16 full-attention layers)	Q4_K (1 Q5_K)	Q8_0
everything else	unchanged (attn_qkv / attn_gate / ssm_out / attn_q / attn_output mostly Q4_K, output Q6_K, token_embd Q4_K)

The imatrix was computed on the official Q8_0 with llama-imatrix -c 2048 --parse-special, over about 442k tokens. The calibration mix has three parts:

128 of ThinkingCap's own chat-formatted reasoning traces (64 prompts x 2 seeds, xhigh effort, <think> included);
450k characters of wikitext-2 train;
350k characters of Python/C source.

How the mix was chosen

Qwen3.8-27B is a hybrid model: 48 gated linear-attention layers and 16 full-attention layers. Starting from Q4_K_S (from f16, same imatrix), I changed one tensor group at a time. Each variant was measured on 3 texts, and I ranked the changes by KLD gain per GiB added.

Change vs Q4_K_S	+GiB	mean KLD change (3 texts)	per GiB
ssm_alpha/beta -> Q8_0	+0.011	-1.3 %	-121 %/GiB
attn_k/v -> Q8_0	+0.089	-6.8 %	-77 %/GiB
ffn_down -> Q5_K	+0.592	-15.1 %	-26 %/GiB
attn_gate + ssm_out -> Q5_K	+0.352	-8.8 %	-25 %/GiB
attn_q + attn_output -> Q5_K	+0.187	-4.4 %	-23 %/GiB
attn_qkv -> Q5_K	+0.277	-6.1 %	-22 %/GiB
token_embd -> Q6_K	+0.306	-1.8 %	-6 %/GiB
ffn_gate/up -> IQ4_XS	-0.337	+1.6 %	cheapest place to save

Moving gate/up to IQ4_XS costs almost nothing, and the space it frees pays for the most efficient upgrades.

Quality: KL divergence vs the official Q8_0

llama-perplexity --kl-divergence, 2048-token chunks, on 3 texts that the imatrix never saw:

wiki: wikitext-2 test, 16 chunks;
code: Python source code, 8 chunks;
traces: 32 held-out ThinkingCap reasoning traces from 16 prompts that were never used for calibration, 16 chunks.

Quant	Size (GiB)	KLD wiki	KLD code	KLD traces	top-1 same (traces)
Q4_K_S (from f16, this imatrix)	14.74	0.0296	0.0683	0.0140	96.7 %
Q4_K_S (from f16, mradermacher imatrix)	14.74	0.0286	0.0719	0.0141	96.8 %
mradermacher i1-Q4_K_S	14.74	0.0281	0.0727	0.0140	96.8 %
mix-b (same size as Q4_K_S)	14.73	0.0257	0.0669	0.0116	96.7 %
MixC (this repo)	15.09	0.0219	0.0534	0.0103	96.9 %
official Q4_K_M	16.25	0.0211	0.0552	0.0109	97.3 %

Two honest notes:

The reasoning-trace imatrix is not a win by itself. Against mradermacher's imatrix with the same recipe, it is -5 % on code, +3 % on wiki and even on traces. The gain comes from the tensor mix.
The reference is Q8_0, not BF16. BF16 did not fit in 32 GB RAM + 24 GB VRAM. Every quant here was made from f16, so none of them inherits the reference's rounding.

Tasks (sanity check, not a leaderboard)

xhigh reasoning, 262k context, temperature 1.0, 12 AIME 2025 problems + 15 HumanEval+ problems. Script: study/bench_tasks.py.

Quant	Score	AIME 2025	HumanEval+	reasoning tokens
MixC	24/27	11/12	13/15	275k
Q4_K_S (from Q8_0)	23/27	10/12	13/15	272k

The only difference is a single AIME problem. With n=27 at temperature 1.0, this is within noise. It shows no reasoning regression, not an improvement.

Speed and memory (RX 7900 XTX, Windows, Vulkan)

llama.cpp master with ggml-org/llama.cpp#29679, llama-bench, no speculation:

Quant	prompt 512	generation (tg128)
Q4_K_S	792 t/s	42.9 t/s
MixC	824 t/s	41.6 t/s

With 262k context, 2 slots, unified q4_0 KV and n-gram + embedded MTP (n=2), the server takes 20.85 GiB (23.47 GiB used on the card): it fits on 24 GB. The MTP head is kept (blk.64) and works with --spec-type draft-mtp.

AMD / Vulkan warning: on RDNA3 with the AMD proprietary driver, llama.cpp builds before #29679 are very slow on 5-7 token batches (speculative decoding, multi-slot). Workarounds that disable MMVQ for those batch sizes are even worse with IQ4_XS: 147 ms vs 32 ms for a 6-token batch. Use a build that includes the fix, or no workaround at all.

Reproduce

Everything is in study/:

calib_prompts.jsonl and gen_traces.py: prompts and reasoning-trace generation;
build_calib.py: builds the calibration text and the held-out split;
Run-QuantPipeline.ps1: imatrix, KLD references, variants and competitors;
Run-ServeMix.ps1: 262k VRAM and speed check;
Run-TaskBench.ps1 and bench_tasks.py: task benchmark;
data/: all raw results (kld_results.jsonl, serve_results.jsonl, tasks_results.jsonl) and the calibration traces.

The scripts use Windows paths from my machine: adjust them to yours.

Limitations

Tested on one GPU (RX 7900 XTX, AMD proprietary Windows driver 26.8.1). CUDA, Metal and RADV speeds are not measured.
KLD uses about 80k tokens of text (llama-perplexity scores the second half of each chunk, so about 40k scored tokens), and the tasks are n=27. Treat small gaps (a few %) as noise.
The mix was tuned for a 262k-on-24 GB budget. mix-b is the choice if you need exactly the Q4_K_S size (numbers above; not uploaded).

License

ThinkingCap is (c) 2026 BottleCap AI and licensed under the PolyForm Small Business License 1.0.0, with an additional personal-use permission (see LICENSE). Commercial use by organizations that are not a "Small Business" needs a license from BottleCap AI. It is derived from Qwen materials licensed under Apache-2.0 (see NOTICE and LICENSE-Apache-2.0-Qwen.txt). This quantization is a change/new work redistributed under the same terms.

Credits

BottleCap AI for ThinkingCap, and the Qwen team for Qwen3.8-27B.
mradermacher for the reference i1 quants and imatrix.
llama.cpp / ggml.
The study scripts and this card were prepared with AI assistance (Claude). The measurements are real runs on my hardware, and the raw data is in study/data.

Downloads last month: 59

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Makdok/ThinkingCap-Qwen3.8-27B-MixC-GGUF

Base model

Qwen/Qwen3.8-27B

Finetuned

bottlecapai/ThinkingCap-Qwen3.8-27B

Quantized

(38)

this model