Instructions to use thinkingmachines/Inkling-2-Mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingmachines/Inkling-2-Mini with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-2-Mini") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-2-Mini") model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-2-Mini", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingmachines/Inkling-2-Mini with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingmachines/Inkling-2-Mini" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/thinkingmachines/Inkling-2-Mini
- SGLang
How to use thinkingmachines/Inkling-2-Mini with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use thinkingmachines/Inkling-2-Mini with Docker Model Runner:
docker model run hf.co/thinkingmachines/Inkling-2-Mini
Multi-GPU llama.cpp benchmarks — 2× RTX 5060 Ti 16GB: pp/tg ladder, max context up to 393k, parallel tests up to 64k (UD-Q4_K_XL)
Inkling 2 Mini on llama.cpp — multi-GPU benchmark results (2× RTX 5060 Ti 16 GB)
We benchmarked Inkling-2-Mini-UD-Q4_K_XL (Mainline_Support variant) on mainline
llama.cpp across two 16 GB consumer GPUs and published the full reproducible setup
(docker compose, patched image, scripts, raw data).
Setup: 2× RTX 5060 Ti 16 GB (Blackwell sm_120, layer-split + auto-fit) · Ryzen 5 7500F ·
llama.cpp @ 2a28df0f9bd5 (inkling2_loop layout #32694 + scale-reshare fix #32704 — the fix is
required, otherwise loop passes 2/3 lose their per-expert NVFP4 scales) · flash-attn on ·
KV q8_0 · ubatch 256 · 12 CPU threads · alias inkling-2-mini
Depth ladder (fresh requests, temp 0, 128 gen)
| Prompt depth | pp (t/s) | tg (t/s) |
|---|---|---|
| 2.6k | 475 | 22.5 |
| 8k | 459 | 23.0 |
| 16k | 419 | 22.0 |
| 32k | 360 | 23.0 |
| 64k | 280 | 20.2 |
| 96k | 225 | 19.7 |
| 128k | 190 | 17.2 |
Max context (per-slot, np=1)
| ctx | KV cache | ubatch | VRAM GiB/GiB | tg (t/s) | Note |
|---|---|---|---|---|---|
| 131,072 | q8_0 | 256 | 13.3 / 14.6 | 24.7 | comfortable default |
| 262,144 | q4_0 | 128 | 13.3 / 14.6 | 24.7 | full speed at 256k! |
| 393,216 | q4_0 | 128 | 14.8 / 14.8 | 17.6 | tight split, still runs |
| 524,288 | q4_0 | ≤128 | — | — | OOM |
Parallel requests (np=4 → per-slot KV, 4× 32k slots)
- 4 concurrent generations (~2.6k prompt each): tg fairly time-sliced at ~9 t/s per stream,
aggregate ≈ 22.5 tok/s ≈ single-stream — decode is routing/loop-bound, not bandwidth-bound - 4× ~14.5k prompts (64k combined prefill): aggregate PP 485 tok/s, +23 % vs sequential
(batched prefill + MoE activation pay off — unlike our dense 27B baseline)
Multi-GPU findings worth knowing
-sm tensoris not implemented forinkling2_loop— layer-split is the only multi-GPU path- Setting
TENSOR_SPLITorN_GPU_LAYERSaborts the auto-fit → the loader tries to allocate
the whole 16 GB model on CUDA0 → OOM. Leave both unset. - The MoE compute buffer scales with context: ubatch 512 only fits up to ~64k ctx;
131k needs 256, 262k needs 128 llama-benchhas the same monolithic-allocation problem on multi-GPU → we benchmarked via
the server API (scripts included)- tg ≈ 22–23 t/s is on par with a dense 27B on the same rig: the loop runs the middle 12
layers 3× (48 applications/token, ~80 % routing/NVFP4-dequant overhead), so VRAM is saved but
compute isn't. DSpark speculative decoding (PR #32218) should be the big lever once it lands.
Links
- 🐳 docker-compose.yml (the exact benchmarked stack, ready to run):
https://github.com/accelerando-developers/inkling-2-mini-llamacpp/blob/main/docker-compose.yml
— patched Dockerfile (llama.cpp @2a28df0f, CUDA sm_120):
https://github.com/accelerando-developers/inkling-2-mini-llamacpp/blob/main/docker/Dockerfile - 📊 Animated architecture comparison (data-flow of this model vs. Qwen3.8-27B dense vs.
Maxwell-V1-Flash MoE — where the bytes flow, when, and where the time is lost):
https://accelerando-developers.github.io/inkling-2-mini-llamacpp/ - 🧾 Full repo with all benchmark scripts + raw data:
https://github.com/accelerando-developers/inkling-2-mini-llamacpp - 📡 Model page on Open LLM Radar: https://open-llm-radar.com/models/inkling-2-mini
- Mainline_Support GGUFs & build instructions:
https://huggingface.co/unsloth/Inkling-2-Mini-GGUF/blob/main/Mainline_Support/README.md - llama.cpp PRs: #32694 (layout) · #32704 (scale-reshare fix) · #32218 (DSpark)
Questions, corrections or results from other multi-GPU rigs are very welcome!