sm_120 support for dflash/speculative decoding

#9
by sharon8811 - opened

Is there going to be a support for speculative decoding for sm_120 gpus?

I also noticed this problem when attempting the vllm recipe on RTX Pro 6000.
I had my agent (running Qwen 3.8 Flash) investigate and solve the problem. Here's the information:

  1. Symptom: the DSpark drafter crashed on RTX PRO 6000 Blackwell (sm_120) at the key-normalization line in the hand-written drafter kernels.
  2. Root cause part one: the shipped kernels carry no sm_120 cubins β€” the fa3 library loaded by the drafter's KV-cache write carries only sm_75 binaries.
  3. Root cause part two: FlashInfer's precompiled RMSNorm has no native switch β€” patched with plain torch fp32 math.
  4. Root cause part three: the low-M matrix-multiply path in linear.py picks FlashInfer CuTe-DSL when use_sm100_kernels β€” same class of problem.
  5. Isolated probes (norm class, argmax, reshape at real shapes) all passed β€” the failure was at library-load time, not the math.
  6. Nsight Systems profiling recorded a kernel-image miss at library-load time, pointing at the sm_75-only fa3 library.
  7. Fix part one: patch the container's RMSNorm with plain torch fp32 math.
  8. Fix part two: swap in the official vLLM 0.34.0 core extension (_C_stable_libtorch.abi3.so β€” 69 sm_120 binaries, including the reshape and cache kernels).
  9. Fix part three: restore the ENTRYPOINT (vllm serve) β€” the prep container had baked in "sleep".
  10. Verified after relaunch: DSpark live, output on a 128k prompt: prose: 318 t/s, python code: 410 tok/s, roughly 3x the drafter-off baseline.

I've created a github repo with a simple script to patch the vllm docker image.
See: https://github.com/Steven0236/inkling2mini-sm120-speculator-fix
Enjoy!

thank you i will try it

For anyone on SGLang instead of vLLM: the SGLang cookbook's rtx6000 DSpark recipe ran on an RTX PRO 6000 Blackwell (sm_120) as printed, with the stock lmsysorg/sglang:inkling-2-mini image (sha256:efac853b46811ccb760545b7b29e7119bdf907537ccf6e863fac3011fe8b98de), no patches.

  • Chat (sglang.benchmark.serving, 12 ShareGPT prompts, 1024 output tokens, one request at a time): 231.3 output tok/s with DSpark, 159.5 with the cookbook's Balanced recipe (no drafter). End-to-end, prefill included.
  • Mean accept length 2.71 on chat, 2.27–2.38 on 20k–120k-token prompts (--speculative-num-draft-tokens 8).
  • FP8 KV (--kv-cache-dtype fp8_e4m3) fails with --attention-backend fa4: q.dtype torch.bfloat16 != k.dtype torch.float8_e4m3fn in _flash_attn_fwd during CUDA graph capture. bf16 KV works.

One card, one run per config. Flags, logs and the rest of the numbers: https://github.com/adcostanza/benchmarks/tree/main/inkling2-mini-vs-qwen38-27b

Sign up or log in to comment