Sakura

Sakura ยท Inkling 2 Mini (GGUF)

Independent community quantization of thinkingmachines/Inkling-2-Mini. Not an official release of Thinking Machines.

Needs llama.cpp b12600 or newer. That is the release these files were made and tested with; it contains the looped Inkling 2 Mini architecture (pull request #32694). Older builds, and Ollama or LM Studio versions that bundle an older llama.cpp, will not load these files (see "Use it" below).

What is this

Inkling 2 Mini is a multimodal looped mixture-of-experts model: 26.4 B parameters, 3.4 B active per token, 128 experts with 8 active, image input, up to 1 M tokens of context. The files here are smaller versions made for local use. The vision projector (mmproj) is included for image input.

Which file

File Quant Size Mean KLD vs Q8_0 Same top token
Sakura-Inkling-2-Mini-Q4_K_M.gguf Q4_K_M 17.48 GB 0.0228 94.49 %
Sakura-Inkling-2-Mini-Q5_K_M.gguf Q5_K_M 19.91 GB 0.0128 95.28 %
Sakura-Inkling-2-Mini-Q6_K.gguf Q6_K 23.49 GB 0.0076 96.53 %

Rule of thumb: Q4_K_M if memory is tight, Q5_K_M as the balanced choice, Q6_K for the closest result. The KLD values compare each file with ggml-org's Q8_0 file on a short held-out text (5 chunks of 256 tokens: English, code, German, Chinese). A small sample: it says little about long contexts, other languages or code quality, and the reference is Q8_0, not the original model.

Image input: use mmproj-Inkling-2-Mini-BF16.gguf (or the smaller Q8_0 one) next to the model file.

Use it

llama.cpp (release b12600 or newer):

# download a llama.cpp release b12600 or newer from https://github.com/ggml-org/llama.cpp/releases
# (or build it: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build -j)
llama-server -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M -c 8192 --port 8080
# or with local files, including image input:
llama-server -m Sakura-Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf -c 8192 --port 8080

Then open http://127.0.0.1:8080 or use the OpenAI-compatible endpoint http://127.0.0.1:8080/v1.

Ollama (not tested by the author; needs an Ollama version that includes the inkling2_loop architecture):

ollama run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M

or with a Modelfile next to the downloaded file:

FROM ./Sakura-Inkling-2-Mini-Q4_K_M.gguf
PARAMETER num_ctx 8192
ollama create inkling2-mini -f Modelfile && ollama run inkling2-mini

LM Studio (not tested by the author; needs a LM Studio version whose llama.cpp runtime includes the inkling2_loop architecture): open the Discover tab, search for webmp3/Sakura-Inkling-2-Mini-GGUF, download the file you want, load it with GPU offload as high as your memory allows and a context length of 8192. For image input, keep the mmproj file in the same folder as the model file.

Reasoning depth is set by the model's reasoning_effort (none to max), see the original model card.

What was and was not measured

  • Measured: mean KLD and top-token agreement against Q8_0 (table above), a short greedy generation test for every file.
  • Not measured: benchmarks, long-context behaviour, image input with these files, speed, GPU backends, agreement with the original checkpoint. No quality claim beyond the numbers above.
  • The 4-bit experts of layers 0 to 19 come from NVFP4 in the original checkpoint, a different 4-bit format than the K-quants here, so the 4-bit files carry a second rounding step. How much that costs was not measured against the original.

Sources and license

Verification details
  • Original revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb; ggml-org repository revision db61c5b492a6a034d986328d28274af0a439aa6c; BF16 file SHA-256 7074c9762e6769a608c7b23960a7fd2444880b1174a3910149749f6d01389aec; matrix SHA-256 553662ef0d5e3d047e5753bf8275c26bbdd4306c60ddf6dc9f46c7f49f19452a.
  • Matrix: according to the ggml-org card computed on the BF16 file with bartowski's v6 calibration data, conversations rendered through the model's chat template. It has entries for 224 of the 226 16-bit weight tensors of the BF16 file; the other 2 (token embedding and output) were quantized without matrix statistics.
  • Made with llama-quantize --imatrix imatrix.gguf from the llama.cpp release b12600 (revision 29b52d398b294509746ec0a3f28948a3107c0213, CPU build). Every file was checked: its tensor types equal the plan of the same tool's --dry-run, it loads and generates text with that build, its KLD against Q8_0 was measured, and each upload was read back from the Hub by SHA-256.
Projector file (unchanged from ggml-org) Size SHA-256
mmproj-Inkling-2-Mini-BF16.gguf 642 MB e914997fe07fb2066bfdce855ec7275c15bc2b624e5423149f1027839593d9cb
mmproj-Inkling-2-Mini-Q8_0.gguf 346 MB 233287272ec5ac39b4168677aba54a4c024f202410f5629fac1830e1059f8abb
  • Sakura-Inkling-2-Mini-Q4_K_M.gguf: 185a9d92c2d0a76e7e21c69bb43f6bddf3601fc7087f1cbe635ff56d107b6e20
  • Sakura-Inkling-2-Mini-Q5_K_M.gguf: 72fbfccd9561a5a08c29de03896f0ad1af75fdb8c889b8da967aa8c922dd1361
  • Sakura-Inkling-2-Mini-Q6_K.gguf: 3955552fb2fb0de471070b83e5f8d82274b25be9ddc977040c9765bf1b4f9d73
ไธญๆ–‡่ฏดๆ˜Ž ยท ๆจฑ่Šฑ Sakura๏ผˆ็‚นๅ‡ปๅฑ•ๅผ€๏ผ‰

้œ€่ฆ llama.cpp b12600 ๆˆ–ๆ›ดๆ–ฐ็‰ˆๆœฌใ€‚ ่ฟ™ๆ˜ฏๅˆถไฝœๅนถๆต‹่ฏ•่ฟ™ไบ›ๆ–‡ไปถๆ‰€็”จ็š„็‰ˆๆœฌ๏ผŒๅŒ…ๅซๅพช็Žฏๆžถๆž„ Inkling 2 Mini๏ผˆๆ‹‰ๅ–่ฏทๆฑ‚ #32694๏ผ‰ใ€‚ๆ›ดๆ—ง็š„็‰ˆๆœฌ๏ผŒไปฅๅŠๅ†…็ฝฎๆ—ง็‰ˆ llama.cpp ็š„ Ollama ๆˆ– LM Studio๏ผŒๆ— ๆณ•ๅŠ ่ฝฝ่ฟ™ไบ›ๆ–‡ไปถใ€‚

  • ๅŽŸๅง‹ๆจกๅž‹๏ผšthinkingmachines/Inkling-2-Mini๏ผŒApache-2.0ใ€‚ๆ–‡ไปถ็”ฑ ggml-org ็š„ BF16 GGUF๏ผˆggml-org/Inkling-2-Mini-GGUF๏ผ‰้‡ๅŒ–่€Œๆฅ๏ผŒๆˆ‘ๆฒกๆœ‰่‡ชๅทฑไปŽๅŽŸๅง‹ๆƒ้‡่ฝฌๆข๏ผ›้‡่ฆๆ€ง็Ÿฉ้˜ตไธŽ่ง†่ง‰ๆŠ•ๅฝฑๆ–‡ไปถ mmproj ๅ‡ไธบ ggml-org ็š„๏ผŒๅŽŸๆ ทๅค็”จใ€‚
  • ๆ–‡ไปถ๏ผšQ4_K_M๏ผˆๅ†…ๅญ˜็ดงๅผ ๆ—ถ๏ผ‰ใ€Q5_K_M๏ผˆๅ‡่กก๏ผ‰ใ€Q6_K๏ผˆๆœ€ๆŽฅ่ฟ‘ๅŽŸๆจกๅž‹๏ผ‰ใ€‚ๅ›พๅƒ่พ“ๅ…ฅ่ฏทๆŠŠ mmproj ๆ–‡ไปถๆ”พๅœจๆจกๅž‹ๆ–‡ไปถๆ—ใ€‚
  • llama.cpp๏ผˆb12600 ๆˆ–ๆ›ดๆ–ฐ็‰ˆๆœฌ๏ผ‰๏ผš
# download a llama.cpp release b12600 or newer from https://github.com/ggml-org/llama.cpp/releases
# (or build it: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build -j)
llama-server -hf webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M -c 8192 --port 8080
# or with local files, including image input:
llama-server -m Sakura-Inkling-2-Mini-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf -c 8192 --port 8080

็„ถๅŽๆ‰“ๅผ€ http://127.0.0.1:8080๏ผŒๆˆ–ไฝฟ็”จๅ…ผๅฎน OpenAI ็š„ๆŽฅๅฃ http://127.0.0.1:8080/v1ใ€‚

  • Ollama๏ผˆไฝœ่€…ๆœชๆต‹่ฏ•๏ผ›้œ€่ฆๅŒ…ๅซ inkling2_loop ๆžถๆž„็š„ Ollama ็‰ˆๆœฌ๏ผ‰๏ผšollama run hf.co/webmp3/Sakura-Inkling-2-Mini-GGUF:Q4_K_M๏ผŒๆˆ–ๅœจๅทฒไธ‹่ฝฝๆ–‡ไปถๆ—ไฝฟ็”จ Modelfile๏ผˆFROM ./Sakura-Inkling-2-Mini-Q4_K_M.gguf๏ผŒPARAMETER num_ctx 8192๏ผ‰ใ€‚
  • LM Studio๏ผˆไฝœ่€…ๆœชๆต‹่ฏ•๏ผ›้œ€่ฆๅ…ถ llama.cpp ่ฟ่กŒๆ—ถๅŒ…ๅซ inkling2_loop ๆžถๆž„็š„็‰ˆๆœฌ๏ผ‰๏ผšๅœจ Discover ๆ ‡็ญพ้กตๆœ็ดข webmp3/Sakura-Inkling-2-Mini-GGUF๏ผŒไธ‹่ฝฝๆ–‡ไปถ๏ผŒGPU ๅธ่ฝฝๆŒ‰ๅ†…ๅญ˜ๅฐฝ้‡่ฐƒ้ซ˜๏ผŒไธŠไธ‹ๆ–‡้•ฟๅบฆ 8192๏ผ›ๅ›พๅƒ่พ“ๅ…ฅๆ—ถๆŠŠ mmproj ๆ–‡ไปถๆ”พๅœจๅŒไธ€ๆ–‡ไปถๅคนใ€‚
  • ๆ ก้ชŒ๏ผšๅผ ้‡็ฑปๅž‹ไธŽๅŒไธ€ๅทฅๅ…ท --dry-run ็š„่ฎกๅˆ’ไธ€่‡ด๏ผ›ๆ–‡ไปถๅฏๅŠ ่ฝฝๅนถ็”Ÿๆˆๆ–‡ๆœฌ๏ผ›ๅทฒๆต‹้‡็›ธๅฏน Q8_0 ็š„ KLD๏ผ›ๆฏๆฌกไธŠไผ ๅŽๆŒ‰ SHA-256 ๅ›ž่ฏปใ€‚
  • ๅทฒๆต‹้‡๏ผšๅ„ๆ–‡ไปถ็›ธๅฏน Q8_0 ็š„ๅนณๅ‡ KLD ไธŽ้ฆ–้€‰่ฏๅ…ƒไธ€่‡ด็އ๏ผˆไธ€ๆฎตๅพˆ็Ÿญ็š„็•™ๅ‡บๆ–‡ๆœฌ๏ผŒ5 ไธชๅ—๏ผŒๆฏๅ— 256 ไธช่ฏๅ…ƒ๏ผŒ่งไธŠ่กจ๏ผ‰ใ€‚ๆœชๆต‹้‡๏ผšๅŸบๅ‡†ๆต‹่ฏ•ใ€้•ฟไธŠไธ‹ๆ–‡ใ€ไฝฟ็”จ่ฟ™ไบ›ๆ–‡ไปถ็š„ๅ›พๅƒ่พ“ๅ…ฅใ€้€Ÿๅบฆใ€GPU ๅŽ็ซฏใ€ไธŽๅŽŸๅง‹ๆฃ€ๆŸฅ็‚น็š„ไธ€่‡ดๆ€งใ€‚ๅŽŸๅง‹ๆฃ€ๆŸฅ็‚น็ฌฌ 0 ่‡ณ 19 ๅฑ‚็š„ 4 ไฝไธ“ๅฎถๆฅ่‡ช NVFP4๏ผŒไธŽ่ฟ™้‡Œ็š„ K ้‡ๅŒ–ๆ ผๅผไธๅŒ๏ผŒๅ› ๆญคๅคšไบ†ไธ€ๆฌก่ˆๅ…ฅ๏ผŒไปฃไปทๆœชไธŽๅŽŸๅง‹ๆจกๅž‹ๆฏ”่พƒใ€‚
Downloads last month
477
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

6-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for webmp3/Sakura-Inkling-2-Mini-GGUF

Quantized
(158)
this model