Instructions to use webmp3/Sakura-Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Use Docker
docker model run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "webmp3/Sakura-Inkling-2-Mini-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "webmp3/Sakura-Inkling-2-Mini-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
- Ollama
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with Ollama:
ollama run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with Docker Model Runner:
docker model run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
- Lemonade
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Sakura-Inkling-2-Mini-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use webmp3/Sakura-Inkling-2-Mini-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Sakura ยท Inkling 2 Mini (GGUF)
Independent community quantization of thinkingmachines/Inkling-2-Mini. Not an official release of Thinking Machines.
Needs llama.cpp b12600 or newer. That is the release these files were made and tested with; it contains the looped Inkling 2 Mini architecture (pull request #32694). Older builds, and Ollama or LM Studio versions that bundle an older llama.cpp, will not load these files (see "Use it" below).
What is this
Inkling 2 Mini is a multimodal looped mixture-of-experts model: 26.4 B parameters, 3.4 B active per token, 128 experts with 8 active, image input, up to 1 M tokens of context. The files here are smaller versions made for local use. The vision projector (mmproj) is included for image input.
Which file
| File | Quant | Size | Mean KLD vs Q8_0 | Same top token |
|---|---|---|---|---|
Sakura-Inkling-2-Mini-Q4_K_M.gguf |
Q4_K_M | 17.48 GB | 0.0228 | 94.49 % |
Sakura-Inkling-2-Mini-Q5_K_M.gguf |
Q5_K_M | 19.91 GB | 0.0128 | 95.28 % |
Sakura-Inkling-2-Mini-Q6_K.gguf |
Q6_K | 23.49 GB | 0.0076 | 96.53 % |
Rule of thumb: Q4_K_M if memory is tight, Q5_K_M as the balanced choice, Q6_K for the closest result. The KLD values compare each file with ggml-org's Q8_0 file on a short held-out text (5 chunks of 256 tokens: English, code, German, Chinese). A small sample: it says little about long contexts, other languages or code quality, and the reference is Q8_0, not the original model.
Image input: use mmproj-Inkling-2-Mini-BF16.gguf (or the smaller Q8_0 one) next to the model file.
Use it
llama.cpp (release b12600 or newer):
# download a llama.cpp release b12600 or newer from https://github.com/ggml-org/llama.cpp/releases
# (or build it: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build -j)
llama-server -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M -c 8192 --port 8080
# or with local files, including image input:
llama-server -m Sakura-Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf -c 8192 --port 8080
Then open http://127.0.0.1:8080 or use the OpenAI-compatible endpoint http://127.0.0.1:8080/v1.
Ollama (not tested by the author; needs an Ollama version that includes the inkling2_loop architecture):
ollama run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M
or with a Modelfile next to the downloaded file:
FROM ./Sakura-Inkling-2-Mini-Q4_K_M.gguf
PARAMETER num_ctx 8192
ollama create inkling2-mini -f Modelfile && ollama run inkling2-mini
LM Studio (not tested by the author; needs a LM Studio version whose llama.cpp runtime includes the inkling2_loop architecture): open the Discover tab, search for webmp3/Sakura-Inkling-2-Mini-GGUF, download the file you want, load it with GPU offload as high as your memory allows and a context length of 8192. For image input, keep the mmproj file in the same folder as the model file.
Reasoning depth is set by the model's reasoning_effort (none to max), see the original model card.
What was and was not measured
- Measured: mean KLD and top-token agreement against
Q8_0(table above), a short greedy generation test for every file. - Not measured: benchmarks, long-context behaviour, image input with these files, speed, GPU backends, agreement with the original checkpoint. No quality claim beyond the numbers above.
- The 4-bit experts of layers 0 to 19 come from NVFP4 in the original checkpoint, a different 4-bit format than the K-quants here, so the 4-bit files carry a second rounding step. How much that costs was not measured against the original.
Sources and license
- Original model: thinkingmachines/Inkling-2-Mini, Apache-2.0, with the original's model acceptable use policy.
- The files were quantized from ggml-org's BF16 GGUF ggml-org/Inkling-2-Mini-GGUF, converted from the original checkpoint by ggml-org (I did not convert from the original weights myself). The importance matrix (
imatrix.gguf) and the vision projector files are ggml-org's, reused unchanged.
Verification details
- Original revision
6e7c8248f75265bdd4fbab5273304243b5d8c2cb; ggml-org repository revisiondb61c5b492a6a034d986328d28274af0a439aa6c; BF16 file SHA-2567074c9762e6769a608c7b23960a7fd2444880b1174a3910149749f6d01389aec; matrix SHA-256553662ef0d5e3d047e5753bf8275c26bbdd4306c60ddf6dc9f46c7f49f19452a. - Matrix: according to the ggml-org card computed on the BF16 file with bartowski's v6 calibration data, conversations rendered through the model's chat template. It has entries for 224 of the 226 16-bit weight tensors of the BF16 file; the other 2 (token embedding and output) were quantized without matrix statistics.
- Made with
llama-quantize --imatrix imatrix.gguffrom the llama.cpp release b12600 (revision29b52d398b294509746ec0a3f28948a3107c0213, CPU build). Every file was checked: its tensor types equal the plan of the same tool's--dry-run, it loads and generates text with that build, its KLD againstQ8_0was measured, and each upload was read back from the Hub by SHA-256.
| Projector file (unchanged from ggml-org) | Size | SHA-256 |
|---|---|---|
mmproj-Inkling-2-Mini-BF16.gguf |
642 MB | e914997fe07fb2066bfdce855ec7275c15bc2b624e5423149f1027839593d9cb |
mmproj-Inkling-2-Mini-Q8_0.gguf |
346 MB | 233287272ec5ac39b4168677aba54a4c024f202410f5629fac1830e1059f8abb |
Sakura-Inkling-2-Mini-Q4_K_M.gguf:185a9d92c2d0a76e7e21c69bb43f6bddf3601fc7087f1cbe635ff56d107b6e20Sakura-Inkling-2-Mini-Q5_K_M.gguf:72fbfccd9561a5a08c29de03896f0ad1af75fdb8c889b8da967aa8c922dd1361Sakura-Inkling-2-Mini-Q6_K.gguf:3955552fb2fb0de471070b83e5f8d82274b25be9ddc977040c9765bf1b4f9d73
ไธญๆ่ฏดๆ ยท ๆจฑ่ฑ Sakura๏ผ็นๅปๅฑๅผ๏ผ
้่ฆ llama.cpp b12600 ๆๆดๆฐ็ๆฌใ ่ฟๆฏๅถไฝๅนถๆต่ฏ่ฟไบๆไปถๆ็จ็็ๆฌ๏ผๅ ๅซๅพช็ฏๆถๆ Inkling 2 Mini๏ผๆๅ่ฏทๆฑ #32694๏ผใๆดๆง็็ๆฌ๏ผไปฅๅๅ ็ฝฎๆง็ llama.cpp ็ Ollama ๆ LM Studio๏ผๆ ๆณๅ ่ฝฝ่ฟไบๆไปถใ
- ๅๅงๆจกๅ๏ผthinkingmachines/Inkling-2-Mini๏ผApache-2.0ใๆไปถ็ฑ ggml-org ็ BF16 GGUF๏ผggml-org/Inkling-2-Mini-GGUF๏ผ้ๅ่ๆฅ๏ผๆๆฒกๆ่ชๅทฑไปๅๅงๆ้่ฝฌๆข๏ผ้่ฆๆง็ฉ้ตไธ่ง่งๆๅฝฑๆไปถ
mmprojๅไธบ ggml-org ็๏ผๅๆ ทๅค็จใ - ๆไปถ๏ผQ4_K_M๏ผๅ
ๅญ็ดงๅผ ๆถ๏ผใQ5_K_M๏ผๅ่กก๏ผใQ6_K๏ผๆๆฅ่ฟๅๆจกๅ๏ผใๅพๅ่พๅ
ฅ่ฏทๆ
mmprojๆไปถๆพๅจๆจกๅๆไปถๆใ - llama.cpp๏ผb12600 ๆๆดๆฐ็ๆฌ๏ผ๏ผ
# download a llama.cpp release b12600 or newer from https://github.com/ggml-org/llama.cpp/releases
# (or build it: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build -j)
llama-server -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M -c 8192 --port 8080
# or with local files, including image input:
llama-server -m Sakura-Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf -c 8192 --port 8080
็ถๅๆๅผ http://127.0.0.1:8080๏ผๆไฝฟ็จๅ
ผๅฎน OpenAI ็ๆฅๅฃ http://127.0.0.1:8080/v1ใ
- Ollama๏ผไฝ่
ๆชๆต่ฏ๏ผ้่ฆๅ
ๅซ
inkling2_loopๆถๆ็ Ollama ็ๆฌ๏ผ๏ผollama run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M๏ผๆๅจๅทฒไธ่ฝฝๆไปถๆไฝฟ็จ Modelfile๏ผFROM ./Sakura-Inkling-2-Mini-Q4_K_M.gguf๏ผPARAMETER num_ctx 8192๏ผใ - LM Studio๏ผไฝ่
ๆชๆต่ฏ๏ผ้่ฆๅ
ถ llama.cpp ่ฟ่กๆถๅ
ๅซ
inkling2_loopๆถๆ็็ๆฌ๏ผ๏ผๅจ Discover ๆ ็ญพ้กตๆ็ดขwebmp3/Sakura-Inkling-2-Mini-GGUF๏ผไธ่ฝฝๆไปถ๏ผGPU ๅธ่ฝฝๆๅ ๅญๅฐฝ้่ฐ้ซ๏ผไธไธๆ้ฟๅบฆ 8192๏ผๅพๅ่พๅ ฅๆถๆmmprojๆไปถๆพๅจๅไธๆไปถๅคนใ - ๆ ก้ช๏ผๅผ ้็ฑปๅไธๅไธๅทฅๅ
ท
--dry-run็่ฎกๅไธ่ด๏ผๆไปถๅฏๅ ่ฝฝๅนถ็ๆๆๆฌ๏ผๅทฒๆต้็ธๅฏนQ8_0็ KLD๏ผๆฏๆฌกไธไผ ๅๆ SHA-256 ๅ่ฏปใ - ๅทฒๆต้๏ผๅๆไปถ็ธๅฏน
Q8_0็ๅนณๅ KLD ไธ้ฆ้่ฏๅ ไธ่ด็๏ผไธๆฎตๅพ็ญ็็ๅบๆๆฌ๏ผ5 ไธชๅ๏ผๆฏๅ 256 ไธช่ฏๅ ๏ผ่งไธ่กจ๏ผใๆชๆต้๏ผๅบๅๆต่ฏใ้ฟไธไธๆใไฝฟ็จ่ฟไบๆไปถ็ๅพๅ่พๅ ฅใ้ๅบฆใGPU ๅ็ซฏใไธๅๅงๆฃๆฅ็น็ไธ่ดๆงใๅๅงๆฃๆฅ็น็ฌฌ 0 ่ณ 19 ๅฑ็ 4 ไฝไธๅฎถๆฅ่ช NVFP4๏ผไธ่ฟ้็ K ้ๅๆ ผๅผไธๅ๏ผๅ ๆญคๅคไบไธๆฌก่ๅ ฅ๏ผไปฃไปทๆชไธๅๅงๆจกๅๆฏ่พใ
- Downloads last month
- 477
4-bit
5-bit
6-bit
Model tree for webmp3/Sakura-Inkling-2-Mini-GGUF
Base model
thinkingmachines/Inkling-2-Mini