vLLM Inkling-2-Mini NVFP4 on a single RTX 3090 (24 GB): working port, 128k context, DSpark

#20
by hiutaky - opened

screenshot-2026-11-21_16-15-57

We got Inkling-2-Mini running on one consumer Ampere GPU (RTX 3090, SM86, TP=1) using the original NVFP4/MXFP8 checkpoint, with no re-quantization. The code is a fork of vLLM PR #69028:

https://github.com/Hiutaky/vllm (branch ampere-port, see INKLING2-RTX3090.md)

What we changed (all in vllm/models/inkling2)

  • Made the Hopper/Blackwell-only paths work on Ampere: PDL gated to SM90+, no TMA below SM90, a Triton fallback for conv_mix, FA2 for the vision tower, minimum capability lowered to 80.
  • Emulated the FP8 (E4M3) KV cache in software, since Ampere has no hardware E4M3 conversions. Our encoder matched torch.float8_e4m3fn on 10M random values.
  • DSpark speculative decoding works (it needs exactly 7 speculative tokens). The draft KV cache stays in bf16.
  • Squeezed VRAM for the KV cache: vision tower removed (text only), prefill chunk of 256, FP8 KV, an explicit KV budget, and the embedding table offloaded to CPU memory (--cpu-offload-params embed_tokens).

Hopper and Blackwell code paths are untouched: every new branch is guarded by a device-capability check.

Results (single RTX 3090)

Decode with DSpark 112–147 tok/s (91 tok/s without it)
Prefill ~2,500–2,700 tok/s up to 33k tokens, ~1,870 tok/s at 126k
Context 131,072 tokens
KV pool 207k tokens with 8 concurrent sequences (224k with 2)
Weights on GPU 17.55 GiB (18.41 GiB without the embedding offload)
Embedding offload cost decode β‰ˆ βˆ’1%, prefill βˆ’1% to βˆ’3%

We ran 8 concurrent requests (2k to 17k tokens each) without errors, and tool calling works with the inkling parser. VRAM is tight (about 23.8 of 24 GB used).

Caveats

  • We have not re-run quality benchmarks (HumanEval+, IFEval) with FP8 KV + DSpark, and FP8 KV is a precision change.
  • The vision tower is disabled in this configuration.
  • We only tested on a 3090 with the vLLM nightly image plus our Python overlay. We have not measured other GPUs.
  • Speculative decoding acceptance was about 20–29% in our tests.

Feedback and tests on other Ampere cards (3080 Ti, 3090 Ti, A5000, A6000) are welcome.

hiutaky changed discussion title from Inkling-2-Mini NVFP4 on a single RTX 3090 (24 GB): working port, 128k context, DSpark to vLLM Inkling-2-Mini NVFP4 on a single RTX 3090 (24 GB): working port, 128k context, DSpark

Was able to reproduce it on a 3090 and, when skipping DSpark, 256k context was able to fit (gpu util 0.98). Have been using it for a little bit to verify it's stable, my harness compacts when passing 75% (~190k ctx) and haven't had an issue.

Prefill peaks between 2.5k and 4.5k depending on depth, token generation speed falls almost perfectly linearly, ~90 t/s at the start, ~64 t/s at 197k used up context. For my personal uses doubling the max kv is more useful than the ~1.4x speedup, hence why I opted out of using DSpark.

Thank you for the work on supporting this model, it does seem like a decent one for limited ram.

Sign up or log in to comment