GRID W2A4 (NVFP4) for Wan2.1-T2V-1.3B and CogVideoX-2B

Post-training quantized transformers with 2-bit weights and 4-bit activations (W2A4) from GRID, packaged so that every quantized linear layer runs as an NVFP4 GEMM on Blackwell tensor cores, together with a one-click benchmark of speed and GPU memory against the BF16 model.

Contents

Path Content
wan2.1-t2v-1.3b/grid_nvfp4.safetensors, config.json 300 quantized linear layers of the Wan2.1-T2V-1.3B transformer
cogvideox-2b/grid_nvfp4.safetensors, config.json 180 quantized linear layers of the CogVideoX-2B transformer
<model>/grid_trellis.safetensors the same weights as 2-bit trellis symbols + E4M3 scales (2.28 bits per weight, see below)
grid_nvfp4.py runtime: replaces the linear layers of a diffusers transformer
bench.py speed and memory of BF16, GRID (NVFP4 weights) and GRID-2bit (trellis weights)
setup_env.sh, run_bench.sh environment (.venv) and one-click benchmark

The other weights (embeddings, adaLN modulation of CogVideoX, output projection, text encoder, VAE) are the BF16 weights of the base models, downloaded from the HuggingFace hub at run time.

Format

For a layer with weight W [out, in], GRID rotates the input with a randomized Hadamard transform R and stores W R as w_gs * w_sf * c: c an FP4 E2M1 value, w_sf an FP8 E4M3 scale per 16 inputs, w_gs an FP32 scale per tensor. These are the values of the GRID 2-bit trellis code (one E4M3 scale per 32 weights, written for both 16-blocks). Activations are quantized to NVFP4 at run time with a static FP32 tensor scale a_gs.

Key Type Content
<layer>.w_fp4 uint8 [out, in/2] E2M1 codes, element 2j in the low nibble (bit 3: sign; bits 0-2: index into 0, .5, 1, 1.5, 2, 3, 4, 6)
<layer>.w_sf float8_e4m3fn [out, in/16] block scales, row-major
<layer>.w_gs, <layer>.a_gs float32 weight and activation tensor scales
<layer>.bias bfloat16 [out] bias
rot_signs.<n> int8 [n] random signs of the rotation of input dimension n (block size: largest power of two dividing n)

With classifier-free guidance, both passes of a step run in one batch (Wan2.1: patched pipeline; CogVideoX: as released), and the conditional input of a layer is quantized as its difference from the unconditional input (inter-branch difference); config.json lists the layers where this is off (Wan2.1 cross-attention K/V).

The 2-bit trellis code is not decoded inside the GEMM here: the file stores the decoded E2M1 values, so the weights take 4.5 bits per weight in memory. The numerics are those of the paper.

grid_trellis.safetensors stores the weights at 2.28 bits per weight: per layer w_sym (uint8 [tiles, 64], the 2-bit symbols of each 16x16 tile, four per byte), w_sf (float8_e4m3fn [out, in/32]), and w_exc / w_exc_fp4 (about 0.8% of the tiles, whose E2M1 codes are stored directly). Decoding it gives exactly the E2M1 values of grid_nvfp4.safetensors (checked for every layer of both models). The grid-2bit variant of the benchmark keeps only this file on the GPU and decodes each layer to NVFP4 operands right before its GEMM (decode_trellis in grid_nvfp4.py, PyTorch ops: about 0.12 s for all 300 layers of Wan2.1 on an H200), so the decoding time is part of the measured step time and only one decoded layer exists at a time.

Benchmark (RTX 5090 or another Blackwell GPU)

bash run_bench.sh                 # first run: creates .venv (PyTorch 2.9 + CUDA 12.8, diffusers 0.39, FlashInfer)
bash run_bench.sh --runs 5        # more runs
bash run_bench.sh --models wan2.1-t2v-1.3b --save_video

Three variants per model: bf16, grid (NVFP4 weights resident, 4.5 bits per weight) and grid-2bit (2-bit trellis weights resident, 2.28 bits per weight). Select with --variants, e.g. bash run_bench.sh --variants bf16,grid-2bit.

Requirements: NVIDIA driver >= 570, python >= 3.10, about 40 GB of disk for the base models. Each model is measured with its default sampling (Wan2.1: 480x832, 81 frames; CogVideoX: 480x720, 49 frames; 50 steps) on one prompt. The text encoder runs once and is removed before the measurement. Results in results/summary.md:

Column Meaning
DiT weights parameters and buffers of the transformer
s / step median time of one denoising step with both CFG passes
speedup BF16 s/step divided by the s/step of the variant
total the whole pipeline call including VAE decoding
peak denoise / avg denoise maximum and mean allocated GPU memory during the denoising steps
peak total maximum allocated GPU memory including VAE decoding

When GRID is loaded, grid_nvfp4.self_test checks the FP4 packing and scale layout of FlashInfer and compares one FP4 GEMM with a reference; the benchmark stops if no FlashInfer GEMM backend (cutlass, cudnn, trtllm) passes. --backend emu runs the same numerics with dequantized BF16 weights on any GPU (slow; for checking only).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for LiqunMa/speed

Finetuned
(52)
this model