GRID W2A4 (NVFP4) for Wan2.1-T2V-1.3B and CogVideoX-2B
Post-training quantized transformers with 2-bit weights and 4-bit activations (W2A4) from GRID, packaged so that every quantized linear layer runs as an NVFP4 GEMM on Blackwell tensor cores, together with a one-click benchmark of speed and GPU memory against the BF16 model.
Contents
| Path | Content |
|---|---|
wan2.1-t2v-1.3b/grid_nvfp4.safetensors, config.json |
300 quantized linear layers of the Wan2.1-T2V-1.3B transformer |
cogvideox-2b/grid_nvfp4.safetensors, config.json |
180 quantized linear layers of the CogVideoX-2B transformer |
<model>/grid_trellis.safetensors |
the same weights as 2-bit trellis symbols + E4M3 scales (2.28 bits per weight, see below) |
grid_nvfp4.py |
runtime: replaces the linear layers of a diffusers transformer |
bench.py |
speed and memory of BF16, GRID (NVFP4 weights) and GRID-2bit (trellis weights) |
setup_env.sh, run_bench.sh |
environment (.venv) and one-click benchmark |
The other weights (embeddings, adaLN modulation of CogVideoX, output projection, text encoder, VAE) are the BF16 weights of the base models, downloaded from the HuggingFace hub at run time.
Format
For a layer with weight W [out, in], GRID rotates the input with a randomized Hadamard transform R and stores
W R as w_gs * w_sf * c: c an FP4 E2M1 value, w_sf an FP8 E4M3 scale per 16 inputs, w_gs an FP32 scale per
tensor. These are the values of the GRID 2-bit trellis code (one E4M3 scale per 32 weights, written for both
16-blocks). Activations are quantized to NVFP4 at run time with a static FP32 tensor scale a_gs.
| Key | Type | Content |
|---|---|---|
<layer>.w_fp4 |
uint8 [out, in/2] |
E2M1 codes, element 2j in the low nibble (bit 3: sign; bits 0-2: index into 0, .5, 1, 1.5, 2, 3, 4, 6) |
<layer>.w_sf |
float8_e4m3fn [out, in/16] |
block scales, row-major |
<layer>.w_gs, <layer>.a_gs |
float32 | weight and activation tensor scales |
<layer>.bias |
bfloat16 [out] |
bias |
rot_signs.<n> |
int8 [n] |
random signs of the rotation of input dimension n (block size: largest power of two dividing n) |
With classifier-free guidance, both passes of a step run in one batch (Wan2.1: patched pipeline; CogVideoX: as
released), and the conditional input of a layer is quantized as its difference from the unconditional input
(inter-branch difference); config.json lists the layers where this is off (Wan2.1 cross-attention K/V).
The 2-bit trellis code is not decoded inside the GEMM here: the file stores the decoded E2M1 values, so the weights take 4.5 bits per weight in memory. The numerics are those of the paper.
grid_trellis.safetensors stores the weights at 2.28 bits per weight: per layer w_sym (uint8 [tiles, 64], the
2-bit symbols of each 16x16 tile, four per byte), w_sf (float8_e4m3fn [out, in/32]), and w_exc / w_exc_fp4
(about 0.8% of the tiles, whose E2M1 codes are stored directly). Decoding it gives exactly the E2M1 values of
grid_nvfp4.safetensors (checked for every layer of both models). The grid-2bit variant of the benchmark keeps
only this file on the GPU and decodes each layer to NVFP4 operands right before its GEMM (decode_trellis in
grid_nvfp4.py, PyTorch ops: about 0.12 s for all 300 layers of Wan2.1 on an H200), so the decoding time is part
of the measured step time and only one decoded layer exists at a time.
Benchmark (RTX 5090 or another Blackwell GPU)
bash run_bench.sh # first run: creates .venv (PyTorch 2.9 + CUDA 12.8, diffusers 0.39, FlashInfer)
bash run_bench.sh --runs 5 # more runs
bash run_bench.sh --models wan2.1-t2v-1.3b --save_video
Three variants per model: bf16, grid (NVFP4 weights resident, 4.5 bits per weight) and grid-2bit (2-bit
trellis weights resident, 2.28 bits per weight). Select with --variants, e.g. bash run_bench.sh --variants bf16,grid-2bit.
Requirements: NVIDIA driver >= 570, python >= 3.10, about 40 GB of disk for the base models. Each model is measured
with its default sampling (Wan2.1: 480x832, 81 frames; CogVideoX: 480x720, 49 frames; 50 steps) on one prompt. The
text encoder runs once and is removed before the measurement. Results in results/summary.md:
| Column | Meaning |
|---|---|
| DiT weights | parameters and buffers of the transformer |
| s / step | median time of one denoising step with both CFG passes |
| speedup | BF16 s/step divided by the s/step of the variant |
| total | the whole pipeline call including VAE decoding |
| peak denoise / avg denoise | maximum and mean allocated GPU memory during the denoising steps |
| peak total | maximum allocated GPU memory including VAE decoding |
When GRID is loaded, grid_nvfp4.self_test checks the FP4 packing and scale layout of FlashInfer and compares one
FP4 GEMM with a reference; the benchmark stops if no FlashInfer GEMM backend (cutlass, cudnn, trtllm) passes.
--backend emu runs the same numerics with dequantized BF16 weights on any GPU (slow; for checking only).
Model tree for LiqunMa/speed
Base model
Wan-AI/Wan2.1-T2V-1.3B-Diffusers