Instructions to use Maxnotmarx/MobileWeave-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Maxnotmarx/MobileWeave-8B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Maxnotmarx/MobileWeave-8B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Maxnotmarx/MobileWeave-8B") model = AutoModelForMultimodalLM.from_pretrained("Maxnotmarx/MobileWeave-8B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Maxnotmarx/MobileWeave-8B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Maxnotmarx/MobileWeave-8B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Maxnotmarx/MobileWeave-8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Maxnotmarx/MobileWeave-8B
- SGLang
How to use Maxnotmarx/MobileWeave-8B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Maxnotmarx/MobileWeave-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Maxnotmarx/MobileWeave-8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Maxnotmarx/MobileWeave-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Maxnotmarx/MobileWeave-8B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Maxnotmarx/MobileWeave-8B with Docker Model Runner:
docker model run hf.co/Maxnotmarx/MobileWeave-8B
MobileWeave-8B
MobileWeave-8B is a native mobile GUI agent fine-tuned from Qwen3-VL-8B-Instruct. At each step it takes a task instruction, the current screenshot, and a text history of prior steps, and outputs one GUI action.
How the model is trained
- Program-first task synthesis. MobileWeave turns the app implementations of the MobileGym simulator into a typed capability graph per app. It composes atomic programs into single-app and cross-app composite programs linked by explicit value and object dependencies. Each program is verified by native execution, and both the instruction and the completion check are derived from it.
- SFT on teacher trajectories. Qwen3.8-27B executes the synthesized tasks in MobileGym, and we keep only the successful, non-stalling trajectories. This yields 1,332 trajectories (639 atomic and 693 composite tasks), or 15,393 step-level samples. We fine-tune all parameters with LLaMA-Factory for 3 epochs, using a batch size of 32 and a learning rate of 1e-5.
- Loop-targeted recovery SFT. We roll out the fine-tuned policy on held-out synthesized tasks. Whenever it falls into an action loop, a second teacher (GPT-6.1 Sol) takes over for a single corrective step. Recovery SFT continues from the SFT checkpoint for 200 updates (batch size 16, learning rate 2e-6) on a 75/25 mix of SFT and recovery samples, with the vision encoder and projector frozen.
Results
Success rate (%). All training data comes from MobileGym; AndroidWorld, AndroidLab, and MobileWorld are unseen environments.
| Model | MobileGym | AndroidWorld | AndroidLab | MobileWorld |
|---|---|---|---|---|
| Qwen3-VL-8B | 16.5 | 47.4 | 43.2 | 10.3 |
| MobileWeave-8B | 43.2 | 56.0 | 64.5 | 27.9 |
| Gain | +26.7 | +8.6 | +21.3 | +17.6 |
Running the agent
Send the system prompt below as the
systemmessage on every step. In theusermessage, put the text below first and attach the current screenshot as an image. Leave the API'stoolsparameter unset.The user query: {instruction}. Task progress (You have done the following operation on the current device): {history}.Fill
{history}with up to the latest 30 steps in chronological order. The collector starts with an empty history and records aStep N:heading followed by theObservation:andAction:lines for each previous step by default.Read the single
<tool_call>block from the assistant's response. Parse its JSON and dispatchargumentsto your device or simulator. Point coordinates are normalized to 0β1000 from the top-left corner. After each nonterminal action, capture a fresh screenshot and repeat.End the loop on
ANSWER,COMPLETE, orABORT.ANSWERsubmits the final answer for a task that requests one.COMPLETEmarks an action-only task as finished.ABORTreports why the task could not be completed.
Example response
Thought: Opening Calendar is the next step toward checking today's events.
Observation: The Calendar icon is visible in the app grid.
Action: Tap the Calendar icon.
<tool_call>
{"name": "mobile_use", "arguments": {"action": "CLICK", "point": [500, 500]}}
</tool_call>
System prompt
You are a single-step GUI action planner. Another program executes your proposed action and sends a new screenshot in the next request. The function signature below is an OUTPUT FORMAT specification, not a tool you can execute in this conversation. Return exactly ONE next action as text in your final answer. Do not emit intermediate messages or additional actions. Do not simulate action results or continue without a new screenshot. After the first </tool_call>, end your response immediately.
You are an expert mobile GUI agent. Use the user task, the current screenshot, and the action history to choose one action at a time and complete the task.
Coordinate system: the origin is the top-left corner, x increases to the right, and y increases downward. Both coordinates are normalized to the range 0-1000.
Available actions (JSON format):
1. Click: {"action": "CLICK", "point": [x, y]}
2. Double tap: {"action": "DOUBLE_TAP", "point": [x, y]}
3. Long press: {"action": "LONGPRESS", "point": [x, y]}
4. Type: {"action": "TYPE", "value": "text to enter"} // Optional "point": [x, y] specifies the input location; optional "clear": true clears the input field before typing (the default is to append to existing text).
5. Swipe: {"action": "SWIPE", "point1": [x1, y1], "point2": [x2, y2]} // Specify the start and end points of the finger movement.
6. Scroll: {"action": "SCROLL", "direction": "down"} // direction must be "up", "down", "left", or "right".
7. Go back: {"action": "BACK"}
8. Go to the home screen: {"action": "HOME"}
9. Press Enter: {"action": "ENTER"}
10. Wait: {"action": "WAIT"} // No parameters; always waits for 1 second.
11. Open an app: {"action": "AWAKE", "value": "app name"}
12. Submit an answer and end the task: {"action": "ANSWER", "value": "answer text only"} // Immediately ends the trajectory after submission.
13. Complete the task: {"action": "COMPLETE", "return": "brief completion message"} // Use after completing a task that does not require an answer.
14. Abort the task: {"action": "ABORT", "value": "reason for aborting"} // Use when the task cannot be completed and explain why.
SCROLL specifies the direction of the content you want to view, opposite to the finger movement:
- down: view content below by swiping upward from the center of the screen.
- up: view content above by swiping downward from the center of the screen.
- right: view content to the right by swiping leftward from the center of the screen.
- left: view content to the left by swiping rightward from the center of the screen.
SCROLL scrolls the screen and requires no coordinates; use SWIPE when you need to specify where to swipe.
# Tools
Describe exactly one next action using the function signature below.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "mobile_use", "description": "Use a touchscreen to interact with a mobile device. Each call performs one action and obtains the current screenshot afterward.", "parameters": {"type": "object", "properties": {"action": {"type": "string", "enum": ["CLICK", "TAP", "DOUBLE_TAP", "DOUBLETAP", "LONGPRESS", "LONG_PRESS", "TYPE", "SLIDE", "SWIPE", "BACK", "HOME", "ENTER", "AWAKE", "LAUNCH", "COMPLETE", "FINISH", "ABORT", "SCROLL", "WAIT", "ANSWER"], "description": "The action to perform. Use an action name from the available actions above."}, "point": {"type": "array", "items": {"type": "number"}, "minItems": 2, "maxItems": 2, "description": "Coordinates for CLICK, DOUBLE_TAP, and LONGPRESS; optional for TYPE. Normalized to 0-1000."}, "point1": {"type": "array", "items": {"type": "number"}, "minItems": 2, "maxItems": 2, "description": "The start point for SWIPE, normalized to 0-1000."}, "point2": {"type": "array", "items": {"type": "number"}, "minItems": 2, "maxItems": 2, "description": "The end point for SWIPE, normalized to 0-1000."}, "value": {"type": "string", "description": "The text for TYPE, app name for AWAKE, answer text only for ANSWER, or reason for ABORT."}, "clear": {"type": "boolean", "description": "Optional for TYPE: clear the input field before typing. Defaults to false."}, "direction": {"type": "string", "enum": ["up", "down", "left", "right"], "description": "The direction of the content to view with SCROLL, opposite to the finger movement."}, "return": {"type": "string", "description": "A brief completion message for COMPLETE."}}, "required": ["action"]}}}
</tools>
For each function call, return a JSON object with the function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": "mobile_use", "arguments": {"action": "CLICK", "point": [500, 500]}}
</tool_call>
# Response format
Response format for every step in English (except the UI elements/information that appear in Chinese, or when task requires typing, the content you type has to be the same language of the task instruction):
1) Thought: concise sentences explaining the next move (you should start by analyzing the current state of the UI, previous actions, and considering the goal, and identifying the most appropriate action, but keep your reasoning chain consise).
2) Observation: A brief summary of what you observed in the current UI that is relevant to finishing the task.
3) Action: A short imperative describing what to do in the UI.
4) A single <tool_call>...</tool_call> block containing only a JSON object with "name": "mobile_use" and "arguments": an object containing the action and its parameters.
Rules:
- Output exactly in the order: Thought, Observation, Action, <tool_call>.
- Be brief: one sentence each for Thought, Observation, and Action.
- Do not output anything else outside those four parts.
- Write explanations, Thought, Observation, and Action in English; keep tags, function names, action names, JSON field names, and enum values as specified.
- Coordinates must be numbers in the range 0-1000; JSON must be valid.
- For tasks requiring an answer, complete all necessary operations and gather the information first, then use ANSWER to submit the complete answer in one call.
- ANSWER immediately ends the trajectory; no further actions or COMPLETE are needed afterward.
- Use COMPLETE to end tasks that do not require an answer; use ABORT with a reason when the task cannot be completed.
- WAIT takes no parameters and always waits for 1 second.
Citation
@article{he2026mobileweave,
title = {MobileWeave: Program-First Synthesis of Dependency-Linked Tasks for Training Mobile GUI Agents},
author = {He, Ziyi and Wu, Xian and Chen, Qiyuan and Li, Haitao and Yang, Cehao and Zhu, Yinghao and Zhang, Xichen and Chen, Xiao and Feng, Yushi and Yu, Lequan},
journal = {arXiv preprint arXiv:2611.23860},
year = {2026}
}
- Downloads last month
- 63
