Instructions to use FutureMLS-Lab/flint-nvfp4-coldgptq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FutureMLS-Lab/flint-nvfp4-coldgptq with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FutureMLS-Lab/flint-nvfp4-coldgptq")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FutureMLS-Lab/flint-nvfp4-coldgptq")
model = AutoModelForCausalLM.from_pretrained("FutureMLS-Lab/flint-nvfp4-coldgptq", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FutureMLS-Lab/flint-nvfp4-coldgptq with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FutureMLS-Lab/flint-nvfp4-coldgptq"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-nvfp4-coldgptq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FutureMLS-Lab/flint-nvfp4-coldgptq

SGLang

How to use FutureMLS-Lab/flint-nvfp4-coldgptq with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FutureMLS-Lab/flint-nvfp4-coldgptq" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-nvfp4-coldgptq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FutureMLS-Lab/flint-nvfp4-coldgptq" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-nvfp4-coldgptq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use FutureMLS-Lab/flint-nvfp4-coldgptq with Docker Model Runner:
```
docker model run hf.co/FutureMLS-Lab/flint-nvfp4-coldgptq
```

flint-nvfp4-coldgptq

NVFP4 quantization of RecursiveSI/flint (revision 2a247be), Recursive's 39.8B-total / ~2.5B-active mixture-of-experts model for coding agents. The checkpoint is 23.5 GB instead of 79.6 GB, so one Blackwell GPU keeps most of its memory for KV cache.

What is quantized

The 12,032 routed experts of MoE layers 2–48 (gate_proj, up_proj, down_proj) are static W4A4 NVFP4: FP4 E2M1 weights in 16-element blocks with FP8 E4M3 block scales and FP32 global scales, in NVIDIA ModelOpt's per-expert layout (weight, weight_scale, weight_scale_2, input_scale).
Every other tensor is the unchanged BF16 source tensor: attention and its learned sinks, routers, shared experts, the dense layers 0–1, norms, embeddings and lm_head.

Evaluation

B300, SGLang 0.5.16 with flint_sglang 0.1.4 and the base repository's serve_b300.sh flags, using the model's default sampling (temperature 1.0, top-p 0.95, top-k 20). Accuracy is the mean over all samples; generation length counts completion tokens, reasoning included.

Benchmark	Accuracy, BF16 → NVFP4	Generation length p50 / p90 (tokens), BF16 → NVFP4
GSM8K, 1,319 questions	96.1 → 96.0	451 / 2,796 → 459 / 3,004
GPQA Diamond, 198 questions × 3 samples	69.2 → 68.9	6,718 / 30,918 → 7,440 / 30,553
AIME 2026, 30 problems × 16 samples	84.4 → 83.8	7,927 / 32,768 → 8,294 / 32,768

GSM8K completions are capped at 16,384 tokens and GPQA and AIME at 32,768. The cap was reached by 8.8% / 7.7% of GPQA samples and 13.3% / 14.2% of AIME samples (BF16 / NVFP4), so AIME's p90 is the cap for both. In GPU memory the NVFP4 weights take 23.6 GB, against 74.2 GB for BF16.

Usage

NVFP4 needs a Blackwell GPU; this checkpoint was validated on B300. Install the Flint SGLang kit from the base repository (sglang/install.sh, or its Dockerfile), then point the kit's serve script at this checkpoint. SGLang reads the ModelOpt NVFP4 configuration from config.json, and the kit's default flags (FA4 attention, flashinfer_trtllm_routed MoE) work unchanged.

hf download FutureMLS-Lab/flint-nvfp4-coldgptq --local-dir flint-nvfp4-coldgptq
hf download RecursiveSI/flint --include "sglang/*" --local-dir flint-kit
MODEL=$PWD/flint-nvfp4-coldgptq bash flint-kit/sglang/serve_b300.sh

Use the chat template and sampling defaults shipped with the checkpoint, as described in the base model card.

License

Apache-2.0, the same license as the base model. See RecursiveSI/flint for the model's intended uses and limitations.

Downloads last month: 338

Safetensors

Model size

21B params

Tensor type

BF16

Model tree for FutureMLS-Lab/flint-nvfp4-coldgptq

Base model

RecursiveSI/flint

Quantized

(96)

this model