Instructions to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="prithivMLmods/Qwen3.6-27B-NVFP4-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("prithivMLmods/Qwen3.6-27B-NVFP4-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
llama cli -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4
# Run inference directly in the terminal:
./build/bin/llama cli -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Use Docker

docker model run hf.co/prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Llama new
LM Studio
Jan

vLLM

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

SGLang

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Ollama:
```
ollama run hf.co/prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4
```
Unsloth Desktop

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4
```

Lemonade

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Run and chat with the model

lemonade run user.Qwen3.6-27B-NVFP4-GGUF-NVFP4

List all available models

lemonade list

Hermes Agent

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use prithivMLmods/Qwen3.6-27B-NVFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "prithivMLmods/Qwen3.6-27B-NVFP4-GGUF:NVFP4" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.6-27B-NVFP4-GGUF

Qwen3.6-27B from Alibaba's Qwen team is a 27B-parameter dense multimodal model with an integrated vision encoder, 64 layers, and a native 262K-token context window extendable to 1M+ via YaRN. It delivers strong agentic coding, repository-level reasoning, and frontend development capabilities, outperforming the previous Qwen3.5-397B-A17B flagship on SWE-bench Verified (77.2), Terminal-Bench (59.3), and SkillsBench (48.2). It also scores 87.8 on GPQA-Diamond and 66.1% on BFCL-V4 for tool calling. Hybrid thinking modes preserve reasoning across multi-turn workflows. It supports vLLM, SGLang, and Unsloth, with GGUF quantization enabling local deployment, and is released under Apache-2.0.

MTP — Speculative Decoding: Yes

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Qwen3.6-27B-NVFP4.gguf	NVFP4	28.2 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 27B parameter model.
Qwen3.6-27B-NVFP4.mmproj.gguf	mmproj	931 MB	Link	Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.6-27B-NVFP4.gguf with vision support on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric	Result
Average Prompt Processing Speed	1,125.92 tokens/s
Average Generation Speed	107.06 tokens/s
Average Wall Time per Image	17.18 s
Total Wall Time (3 Images)	51.5 s
Context Length	16,384 tokens
KV Cache (K)	`q8_0`
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity	96 GB
Peak GPU VRAM Usage	28,217 MiB (27.56 GiB)
Peak GPU Utilization	95%
GPU Layers Offloaded	99 (`-ngl 99`)
Random Seed	42
Speculative Decoding	Enabled (MTP, draft_n_max=2)
Test Runs / Images	3
Average Decode Speed (Minimum)	105.15 tokens/s
Average Decode Speed (Maximum)	108.06 tokens/s

Test Shots

Shot	Task	Prompt Processing (tokens/s)	Generation (tokens/s)	Wall Time
Shot 1	Vision OCR - Image 1	1,138.18	108.06	13.974 s
Shot 2	Vision OCR - Image 2	1,123.95	107.96	17.567 s
Shot 3	Vision OCR - Image 3	1,115.62	105.15	20.006 s
Average	3 OCR images	1,125.92	107.06	17.18 s

Run Inference

# Download a test image
wget -O /path/to/models/input.jpeg \
  "https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/EF8cldKHhUEO3MZPZSD19.jpeg"

# Run Qwen3.6-27B NVFP4 with vision support
/data/llama.cpp/build/bin/llama-cli \
  -m /path/to/models/Qwen3.6-27B-NVFP4.gguf \
  --mmproj /path/to/models/Qwen3.6-27B-NVFP4.mmproj.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  --image /path/to/models/input.jpeg \
  -p "OCR the image."

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the input image, prompt, model, software build, and runtime settings.

Snapshots (3)

License and Attributions

Base Model: Qwen/Qwen3.6-27B: Used for NVFP4 compression and quantization.
Quantization: Performed using llama.cpp by prithivMLmods.
License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-27B base model.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Downloads last month: 359

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for prithivMLmods/Qwen3.6-27B-NVFP4-GGUF

Base model

Qwen/Qwen3.6-27B

Quantized

(754)

this model

Collections including prithivMLmods/Qwen3.6-27B-NVFP4-GGUF