Instructions to use KRAFTON/Raon-1-mini-preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KRAFTON/Raon-1-mini-preview with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KRAFTON/Raon-1-mini-preview", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("KRAFTON/Raon-1-mini-preview", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KRAFTON/Raon-1-mini-preview with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KRAFTON/Raon-1-mini-preview"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KRAFTON/Raon-1-mini-preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KRAFTON/Raon-1-mini-preview

SGLang

How to use KRAFTON/Raon-1-mini-preview with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KRAFTON/Raon-1-mini-preview" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KRAFTON/Raon-1-mini-preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KRAFTON/Raon-1-mini-preview" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KRAFTON/Raon-1-mini-preview",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KRAFTON/Raon-1-mini-preview with Docker Model Runner:
```
docker model run hf.co/KRAFTON/Raon-1-mini-preview
```

Sharing a vLLM(B12X-optimized) Raon-1-mini NVFP4 image and serving guide for DGX Spark (SM121)

by Pilcothink - opened 3 days ago

Discussion

Pilcothink

3 days ago

•

edited 3 days ago

https://github.com/gpdev-Pilcothink/DGX_Spark_vllm_Dockerfile/tree/main/0.32/Raon1-mini

The link above contains the Docker image and an easy vLLM serving guide for Raon-1-mini-preview NVFP4.

2027 크래프톤과 좋은 인연이 되기를 기대하며.., 최적화 및 테스트를 진행한 DGX Spark용 가이드(도커이미지) 및 양자화 모델을 공유합니다.

개인적으로 Raon-1-mini는 게임 NPC에 탑재하기 좋은 한국 모델이라고 생각이 드네요. 여러 NPC를 서버에서 동시에 처리한다면 처리량과 비용 효율도 중요하다고 생각해서, 단일 요청뿐 아니라 동시 요청 성능 개선에도 신경을 썼습니다. 특히 제가 양자화를 진행한 Dflash2-nvfp4 모델은 GB300같은 기기에서도 제대로 작동할텐데, 누군가 해당 기기에서 써보셨으면 좋겠습니다.

프리뷰가 아닌 정식 출시 모델에는 DFlash 대신 MTP 방식의 드래프트가 적용되면 좋겠다는 생각을 했습니다.
MTP의 수락률이 높게 유지되고 NVFP4 양자화까지 적용할 수 있다면, 드래프트 연산 비용을 줄여 서버에서 더 큰 배치를 효율적으로 처리하는 데 도움이 될 것 같습니다.

RTX 5090이나 RTX PRO 6000에서도 테스트해 보고 싶지만 해당 장비가 없어서 매우 아쉽네요 ㅜㅜ (테스트 해줄 사람을 찾고 있습니다.)

What changed

KRAFTON already provides a DGX Spark image for Raon-1-mini-preview NVFP4. I developed a separate image using a newer vLLM version and B12X kernels to explore overall performance improvements on a single DGX Spark.

The image uses:

vLLM 0.32.0
B12X Linear and MoE [https://github.com/local-inference-lab/flashinfer]
A custom B12X_MLA adapter for target MLA decode and speculative verification
FP8 E4M3 target KV cache
A quantized DFlash2 NVFP4 W4A4 draft
A separately quantized NVFP4 W4A4 draft output head
FLASH_ATTN draft attention
4 speculative tokens
Probabilistic drafting and block rejection sampling
Async scheduling and FULL decode CUDA graphs

The DFlash2 draft and its output head were quantized using NVIDIA Model Optimizer.

Models:

Target: KRAFTON/Raon-1-mini-preview-nvfp4
Quantized draft: Pilcothink/Raon-1-mini-preview-dflash2-nvfp4

Performance comparison

The following llama-benchy results compare KRAFTON's original DGX Spark image with my B12X image on a single DGX Spark.

The images use different vLLM implementations and serving configurations, so this is an image-level comparison, not a measurement of one optimization in isolation.

Single-request tests

Test	KRAFTON image	B12X image	Change
pp2048, tg128 run	3,957.45 ± 37.69 tok/s	6,614.65 ± 188.99 tok/s	+67.1%
tg128	57.54 ± 2.71 tok/s	71.95 ± 3.21 tok/s	+25.0%
pp2048, tg1024 run	4,737.45 ± 324.32 tok/s	6,811.57 ± 57.61 tok/s	+43.8%
tg1024	63.10 ± 1.17 tok/s	71.25 ± 4.64 tok/s	+12.9%
pp2048 @ d1024, tg128 run	5,392.41 ± 136.57 tok/s	6,269.53 ± 197.09 tok/s	+16.3%
tg128 @ d1024	51.31 ± 6.72 tok/s	70.25 ± 6.97 tok/s	+36.9%
pp2048 @ d1024, tg1024 run	4,657.61 ± 1,367.24 tok/s	6,146.96 ± 170.35 tok/s	+32.0%
tg1024 @ d1024	58.22 ± 6.58 tok/s	67.23 ± 2.90 tok/s	+15.5%
pp2048 @ d4096, tg128 run	5,900.02 ± 284.13 tok/s	6,541.13 ± 103.36 tok/s	+10.9%
tg128 @ d4096	51.70 ± 4.05 tok/s	74.07 ± 6.89 tok/s	+43.3%
pp2048 @ d4096, tg1024 run	6,165.15 ± 247.25 tok/s	6,506.45 ± 100.87 tok/s	+5.5%
tg1024 @ d4096	54.17 ± 1.63 tok/s	68.02 ± 2.39 tok/s	+25.6%

Concurrent requests

These figures show total output throughput. Concurrency is the number of requests submitted by the benchmark.

Test	KRAFTON image	B12X image	Change
tg256 @ d1024, c1	56.51 ± 2.00 tok/s	78.64 ± 4.06 tok/s	+39.2%
tg256 @ d1024, c10	118.89 ± 2.14 tok/s	179.39 ± 4.41 tok/s	+50.9%
tg256 @ d1024, c50	146.23 ± 2.03 tok/s	254.79 ± 1.56 tok/s	+74.2%

At 50 concurrent requests, total output throughput increased from 146.23 to 254.79 tok/s.

Long context: 100K depth

Metric	KRAFTON image	B12X image	Change
pp2048 @ d100000	4,100.50 ± 7.39 tok/s	3,931.13 ± 181.86 tok/s	−4.1%
tg512 @ d100000	33.23 ± 1.51 tok/s	54.44 ± 5.06 tok/s	+63.8%
Time to first response	24,893.14 ± 44.89 ms	26,021.91 ± 1,205.14 ms	+4.5%

At 100K depth, generation throughput increased from 33.23 to 54.44 tok/s — approximately 63.8% faster — while prefill throughput remained close to the original image.

With MLA, longer contexts require each decode step to read and process more cached data, which can reduce generation speed.

Some earlier long-context runs experienced memory pressure or unexpected engine termination. I am continuing to check long-context stability.

Quality evaluation

For the tool-calling evaluation and its detailed results, please see my NVIDIA Developer Forum post:

https://forums.developer.nvidia.com/t/let-s-optimize-raon-1-mini-preview-nvfp4-for-single-dgx-spark/387340

Looking for SM120 testers

I would love to test this setup on an RTX 5090 or RTX PRO 6000 Blackwell, but I do not have access to either device.

If you have one of these GPUs and would be interested in testing, please contact me or share your results here.

ddwkim

KRAFTON org 3 days ago

Hi @Pilcothink , thank you for putting this together and sharing it! The B12X-optimized image and the NVFP4 DFlash2 draft are really nice work, and we appreciate how carefully you laid out the llama-benchy results, especially the image-level caveat and the notes on long-context stability.

We have RTX PRO 6000 Blackwell and RTX 5090 machines, so we'll try your image on SM120 when we get the chance and share the results here. Thanks also for the suggestion about an MTP-style draft for the non-preview release. We'll take it into account.

Pilcothink

3 days ago

Unfortunately, the current image is still ARM-based and designed exclusively for DGX Spark. I'll build an SM120-compatible image and provide an updated Docker image.

Pilcothink

3 days ago

# 이미지와 모델 다운로드
docker pull pilcothink/vllm_pc:0.32-Raon-1-v4

hf download KRAFTON/Raon-1-mini-preview-nvfp4 \
  --local-dir /path/Model/Raon-1-mini-preview-nvfp4

hf download Pilcothink/Raon-1-mini-preview-dflash2-nvfp4 \
  --local-dir /path/Model/Raon-1-mini-preview-dflash2-nvfp4

/path를 모델을 저장한 경로로 바꿔 실행하세요.

HOST_AI_DIR=/path

docker run --rm -it \
  --gpus all \
  --network host \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v "$HOST_AI_DIR:/workspace" \
  -v "$HOME/.cache/raon-vllm-pc:/root/.cache" \
  -e CUTE_DSL_ARCH=sm_120a \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  --entrypoint vllm \
  pilcothink/vllm_pc:0.32-Raon-1-v4 \
  serve /workspace/Model/Raon-1-mini-preview-nvfp4 \
    --served-model-name Raon-1-mini-preview \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --decode-context-parallel-size 1 \
    --dtype bfloat16 \
    --load-format safetensors \
    --linear-backend b12x \
    --moe-backend b12x \
    --no-enable-flashinfer-autotune \
    --attention-backend B12X_MLA \
    --attention-config '{"flash_attn_version":2}' \
    --kv-cache-dtype fp8_e4m3 \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --async-scheduling \
    --max-model-len 32768 \
    --max-num-seqs 4 \
    --max-num-batched-tokens 2048 \
    --gpu-memory-utilization 0.80 \
    --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --speculative-config '{"method":"dflash","model":"/workspace/Model/Raon-1-mini-preview-dflash2-nvfp4","num_speculative_tokens":4,"draft_sample_method":"probabilistic","rejection_sample_method":"block","attention_backend":"FLASH_ATTN","kv_cache_dtype":"auto","moe_backend":"auto"}' \
    --reasoning-parser raon_1 \
    --tool-call-parser raon_1 \
    --enable-auto-tool-choice \
    --default-chat-template-kwargs '{"enable_thinking":true,"truncate_history_thinking":false}'

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment