Instructions to use thinkingmachines/Inkling-2-Mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingmachines/Inkling-2-Mini with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-2-Mini") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-2-Mini") model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-2-Mini", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingmachines/Inkling-2-Mini with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingmachines/Inkling-2-Mini" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/thinkingmachines/Inkling-2-Mini
- SGLang
How to use thinkingmachines/Inkling-2-Mini with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use thinkingmachines/Inkling-2-Mini with Docker Model Runner:
docker model run hf.co/thinkingmachines/Inkling-2-Mini
Tool calling: model plans an ask_user call, writes a lead-in sentence, then ends its turn without the call
Summary
When a tool for asking the user questions is declared, Inkling-2-Mini often decides in its reasoning to call it, writes one lead-in sentence as text, and then ends its turn. No tool call is emitted. The user sees a reply such as "Sure—before I draft it, could you answer a couple of quick questions?" and nothing else.
I first saw this in Open WebUI 0.13.3, whose built-in tools include ask_user. It reproduces without Open WebUI, on both vLLM and SGLang, with a single declared tool.
Reproduction
The script below sends a 7-message conversation with one tool (ask_user, declared as Open WebUI declares it) to an OpenAI-compatible endpoint and counts what each reply does. It uses only the Python standard library and server-default sampling.
python3 ask_user_repro.py http://127.0.0.1:8080 48
ask_user_repro.py
"""Inkling-2-Mini plans an ask_user tool call, writes a lead-in sentence, and ends its turn.
Usage: python3 ask_user_repro.py http://127.0.0.1:8080 [samples]
Sends one short conversation with a single declared tool to an OpenAI-compatible
server and counts what each reply does. Standard library only.
"""
import collections
import concurrent.futures
import json
import sys
import urllib.request
BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080"
SAMPLES = int(sys.argv[2]) if len(sys.argv) > 2 else 48
MODEL = "thinkingmachines/Inkling-2-Mini"
# The ask_user tool as Open WebUI 0.13.3 declares it.
TOOLS = [{
"type": "function",
"function": {
"name": "ask_user",
"description": "Ask the user clarifying questions before continuing.\n"
"Use this when the next step depends on user intent, preference, "
"or a tradeoff that cannot be inferred safely.",
"parameters": {
"type": "object",
"properties": {
"questions": {
"type": "array",
"items": {"type": "object"},
"description": "1-3 question objects, each with id, header, question, and 2-3 "
"options. Each option needs label and description. List the "
"option you recommend first; the UI labels the first option "
"Recommended.",
},
"allow_other": {
"type": "boolean",
"description": "Whether users may enter a free-form answer instead of "
"choosing one of the options",
"default": True,
},
"timeout_ms": {
"type": "integer",
"description": "How long the browser should keep the prompt open before "
"cancelling it",
"default": 120000,
},
},
"required": ["questions"],
},
},
}]
MESSAGES = [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hello! How can I help you today?"},
{"role": "user", "content": "How are you"},
{"role": "assistant", "content": "I'm doing well, thanks for asking! Ready to help with whatever "
"you'd like—whether it's work, scheduling, notes, or just a "
"question. What's on your mind?"},
{"role": "user", "content": "Can you help me draft a clear, professional work email?"},
{"role": "assistant", "content": "I'd be glad to help! To draft something useful, could you tell "
"me a bit about it?"},
{"role": "user", "content": "Help draft a polite follow-up email to my manager about a delayed report"},
]
FAILED = "planned ask_user in its reasoning, wrote a lead-in, ended the turn without the call"
def sample(_):
body = {"model": MODEL, "messages": MESSAGES, "tools": TOOLS, "max_tokens": 1500}
request = urllib.request.Request(
BASE + "/v1/chat/completions", json.dumps(body).encode(), {"Content-Type": "application/json"}
)
choice = json.loads(urllib.request.urlopen(request, timeout=600).read())["choices"][0]
message = choice["message"]
calls = [call["function"]["name"] for call in message.get("tool_calls") or []]
text = message.get("content") or ""
reasoning = message.get("reasoning") or message.get("reasoning_content") or ""
if "ask_user" in calls:
kind = "called ask_user"
elif len(text) > 400:
kind = "wrote the email draft"
elif "ask_user" in reasoning:
kind = FAILED
else:
kind = "asked a short question in text, no tool planned"
return kind, choice["finish_reason"], text, reasoning
with concurrent.futures.ThreadPoolExecutor(8) as pool:
results = list(pool.map(sample, range(SAMPLES)))
counts = collections.Counter(kind for kind, *_ in results)
for kind, count in counts.most_common():
print(f"{count:3d}/{SAMPLES} {kind}")
for kind, finish_reason, text, reasoning in results:
if kind == FAILED:
print(f"\nExample (finish_reason={finish_reason}):")
print(f" reasoning: ...{reasoning[-300:]!r}")
print(f" content: {text!r}")
break
Results
| Request | Server | Planned the call, never made it |
|---|---|---|
7 messages, ask_user only (the script) |
vLLM | 47 of 96 |
7 messages, ask_user only (the script) |
SGLang | 13 of 48 |
1 message, ask_user only |
vLLM | 6 of 48 |
| 7 messages, 60 tools (Open WebUI's built-in set) | vLLM | 27 of 144 |
| 7 messages, 60 tools (Open WebUI's built-in set) | SGLang | 14 of 96 |
The remaining replies either wrote the email draft or called a tool. The 60 tool definitions were rebuilt from Open WebUI's source, so they approximate what it sends. With no tools declared I did not see the problem in about 60 requests.
Raw output of one failing reply (vLLM /v1/completions on the rendered prompt, special tokens kept, finish_reason: stop):
<|message_model|><|content_thinking|>The user wants help drafting a polite follow-up email to their manager about a delayed report. This is a benign workplace writing task. There are no tools needed for drafting an email—it's a writing task. However, I could ask clarifying questions. The user hasn't provided details: the report name, due date, manager's name, current status, deadline, tone preferences.
The ask_user tool is available and appropriate: "Use this when the next step depends on user intent, preference, or a tradeoff that cannot be inferred safely." I should ask clarifying questions before drafting a professional email. The user's request is vague, so asking for a few details is reasonable.
But note: I don't know if I should draft a generic email instead. The guidance says use ask_user when next step depends on user intent. Drafting an email to a manager depends on details: what's the report, when was it due, current status, manager's name/tone, relationship. I'll ask a concise set of questions (1-3).<|end_message|><|message_model|><|content_text|>I can help with that—it would help if I knew a little more about the situation.<|end_message|>
Sampling ends there. There is no <|content_invoke_tool_json|> block after the text block.
Expected
After deciding to use the tool, the model emits the tool call (with or without a lead-in sentence), or asks its questions in the text itself.
What I ruled out
- The serving stack: vLLM and SGLang both show it, with their own prompt rendering and parsers.
- Output parsing: the raw token stream above has no tool-call block, and in the failing SGLang replies the completion token count equals reasoning plus the short text.
- Speculative decoding and KV cache precision: on vLLM the rate in the 60-tool replay was 8 of 48 with DSpark off and a BF16 KV cache, against 12 of 48 with DSpark on and an FP8 KV cache.
- When the model does emit the call, both servers return it correctly as
tool_calls.
Environment
- Checkpoint revision
ed40e7da625f594fc9c4990f5f66be24873c22d7 - 1x NVIDIA RTX PRO 6000 Blackwell (SM120), one server per card
- vLLM: image
vllm/vllm-openai:inkling-2-mini(sha256:a485440c62099c39659a7b723636073ed726dcb41f94cf6326e16af3a1d6f6bc, vLLM0.1.dev25855+g7cb5dae22),--reasoning-parser inkling --tool-call-parser inkling --enable-auto-tool-choice --kv-cache-dtype fp8, DSpark with 7 draft tokens. DSpark needs a small local patch on this card (the image's draft KV-cache write has no SM12x kernel); the failure is the same with DSpark off. - SGLang: image
lmsysorg/sglang:inkling-2-mini(sha256:421aed3ac11741381eac350bf7b1db28a0cafb6ae49b57ae78fba86a16a32306), the cookbook's RTX PRO 6000 "Balanced" flags:--tp 1 --ep 1 --attention-backend fa4 --dtype bfloat16 --page-size 128 --grammar-backend xgrammar --enable-multimodal --reasoning-parser inkling --tool-call-parser inkling2 - Sampling: server defaults (temperature 1.0 from
generation_config.json), default reasoning effort