Tool calling: model plans an ask_user call, writes a lead-in sentence, then ends its turn without the call

#21
by janabimustafa - opened

Summary

When a tool for asking the user questions is declared, Inkling-2-Mini often decides in its reasoning to call it, writes one lead-in sentence as text, and then ends its turn. No tool call is emitted. The user sees a reply such as "Sure—before I draft it, could you answer a couple of quick questions?" and nothing else.

I first saw this in Open WebUI 0.13.3, whose built-in tools include ask_user. It reproduces without Open WebUI, on both vLLM and SGLang, with a single declared tool.

Reproduction

The script below sends a 7-message conversation with one tool (ask_user, declared as Open WebUI declares it) to an OpenAI-compatible endpoint and counts what each reply does. It uses only the Python standard library and server-default sampling.

python3 ask_user_repro.py http://127.0.0.1:8080 48
ask_user_repro.py
"""Inkling-2-Mini plans an ask_user tool call, writes a lead-in sentence, and ends its turn.

Usage: python3 ask_user_repro.py http://127.0.0.1:8080 [samples]

Sends one short conversation with a single declared tool to an OpenAI-compatible
server and counts what each reply does. Standard library only.
"""
import collections
import concurrent.futures
import json
import sys
import urllib.request

BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080"
SAMPLES = int(sys.argv[2]) if len(sys.argv) > 2 else 48
MODEL = "thinkingmachines/Inkling-2-Mini"

# The ask_user tool as Open WebUI 0.13.3 declares it.
TOOLS = [{
    "type": "function",
    "function": {
        "name": "ask_user",
        "description": "Ask the user clarifying questions before continuing.\n"
                       "Use this when the next step depends on user intent, preference, "
                       "or a tradeoff that cannot be inferred safely.",
        "parameters": {
            "type": "object",
            "properties": {
                "questions": {
                    "type": "array",
                    "items": {"type": "object"},
                    "description": "1-3 question objects, each with id, header, question, and 2-3 "
                                   "options. Each option needs label and description. List the "
                                   "option you recommend first; the UI labels the first option "
                                   "Recommended.",
                },
                "allow_other": {
                    "type": "boolean",
                    "description": "Whether users may enter a free-form answer instead of "
                                   "choosing one of the options",
                    "default": True,
                },
                "timeout_ms": {
                    "type": "integer",
                    "description": "How long the browser should keep the prompt open before "
                                   "cancelling it",
                    "default": 120000,
                },
            },
            "required": ["questions"],
        },
    },
}]

MESSAGES = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hello! How can I help you today?"},
    {"role": "user", "content": "How are you"},
    {"role": "assistant", "content": "I'm doing well, thanks for asking! Ready to help with whatever "
                                     "you'd like—whether it's work, scheduling, notes, or just a "
                                     "question. What's on your mind?"},
    {"role": "user", "content": "Can you help me draft a clear, professional work email?"},
    {"role": "assistant", "content": "I'd be glad to help! To draft something useful, could you tell "
                                     "me a bit about it?"},
    {"role": "user", "content": "Help draft a polite follow-up email to my manager about a delayed report"},
]

FAILED = "planned ask_user in its reasoning, wrote a lead-in, ended the turn without the call"


def sample(_):
    body = {"model": MODEL, "messages": MESSAGES, "tools": TOOLS, "max_tokens": 1500}
    request = urllib.request.Request(
        BASE + "/v1/chat/completions", json.dumps(body).encode(), {"Content-Type": "application/json"}
    )
    choice = json.loads(urllib.request.urlopen(request, timeout=600).read())["choices"][0]
    message = choice["message"]
    calls = [call["function"]["name"] for call in message.get("tool_calls") or []]
    text = message.get("content") or ""
    reasoning = message.get("reasoning") or message.get("reasoning_content") or ""
    if "ask_user" in calls:
        kind = "called ask_user"
    elif len(text) > 400:
        kind = "wrote the email draft"
    elif "ask_user" in reasoning:
        kind = FAILED
    else:
        kind = "asked a short question in text, no tool planned"
    return kind, choice["finish_reason"], text, reasoning


with concurrent.futures.ThreadPoolExecutor(8) as pool:
    results = list(pool.map(sample, range(SAMPLES)))

counts = collections.Counter(kind for kind, *_ in results)
for kind, count in counts.most_common():
    print(f"{count:3d}/{SAMPLES}  {kind}")
for kind, finish_reason, text, reasoning in results:
    if kind == FAILED:
        print(f"\nExample (finish_reason={finish_reason}):")
        print(f"  reasoning: ...{reasoning[-300:]!r}")
        print(f"  content:   {text!r}")
        break

Results

Request Server Planned the call, never made it
7 messages, ask_user only (the script) vLLM 47 of 96
7 messages, ask_user only (the script) SGLang 13 of 48
1 message, ask_user only vLLM 6 of 48
7 messages, 60 tools (Open WebUI's built-in set) vLLM 27 of 144
7 messages, 60 tools (Open WebUI's built-in set) SGLang 14 of 96

The remaining replies either wrote the email draft or called a tool. The 60 tool definitions were rebuilt from Open WebUI's source, so they approximate what it sends. With no tools declared I did not see the problem in about 60 requests.

Raw output of one failing reply (vLLM /v1/completions on the rendered prompt, special tokens kept, finish_reason: stop):

<|message_model|><|content_thinking|>The user wants help drafting a polite follow-up email to their manager about a delayed report. This is a benign workplace writing task. There are no tools needed for drafting an email—it's a writing task. However, I could ask clarifying questions. The user hasn't provided details: the report name, due date, manager's name, current status, deadline, tone preferences.

The ask_user tool is available and appropriate: "Use this when the next step depends on user intent, preference, or a tradeoff that cannot be inferred safely." I should ask clarifying questions before drafting a professional email. The user's request is vague, so asking for a few details is reasonable.

But note: I don't know if I should draft a generic email instead. The guidance says use ask_user when next step depends on user intent. Drafting an email to a manager depends on details: what's the report, when was it due, current status, manager's name/tone, relationship. I'll ask a concise set of questions (1-3).<|end_message|><|message_model|><|content_text|>I can help with that—it would help if I knew a little more about the situation.<|end_message|>

Sampling ends there. There is no <|content_invoke_tool_json|> block after the text block.

Expected

After deciding to use the tool, the model emits the tool call (with or without a lead-in sentence), or asks its questions in the text itself.

What I ruled out

  • The serving stack: vLLM and SGLang both show it, with their own prompt rendering and parsers.
  • Output parsing: the raw token stream above has no tool-call block, and in the failing SGLang replies the completion token count equals reasoning plus the short text.
  • Speculative decoding and KV cache precision: on vLLM the rate in the 60-tool replay was 8 of 48 with DSpark off and a BF16 KV cache, against 12 of 48 with DSpark on and an FP8 KV cache.
  • When the model does emit the call, both servers return it correctly as tool_calls.

Environment

  • Checkpoint revision ed40e7da625f594fc9c4990f5f66be24873c22d7
  • 1x NVIDIA RTX PRO 6000 Blackwell (SM120), one server per card
  • vLLM: image vllm/vllm-openai:inkling-2-mini (sha256:a485440c62099c39659a7b723636073ed726dcb41f94cf6326e16af3a1d6f6bc, vLLM 0.1.dev25855+g7cb5dae22), --reasoning-parser inkling --tool-call-parser inkling --enable-auto-tool-choice --kv-cache-dtype fp8, DSpark with 7 draft tokens. DSpark needs a small local patch on this card (the image's draft KV-cache write has no SM12x kernel); the failure is the same with DSpark off.
  • SGLang: image lmsysorg/sglang:inkling-2-mini (sha256:421aed3ac11741381eac350bf7b1db28a0cafb6ae49b57ae78fba86a16a32306), the cookbook's RTX PRO 6000 "Balanced" flags: --tp 1 --ep 1 --attention-backend fa4 --dtype bfloat16 --page-size 128 --grammar-backend xgrammar --enable-multimodal --reasoning-parser inkling --tool-call-parser inkling2
  • Sampling: server defaults (temperature 1.0 from generation_config.json), default reasoning effort

Sign up or log in to comment