DeepSeek ViT 412M — V4 Flash Vision Exp

Standalone vision backbone extracted from DeepSeek-V4-Flash-Vision-Exp, converted to native timm format without additional training. No language-model weights or trained classification head are included.

The conversion follows timm/deepseek_vit_412m.deepseek_v4_flash_vision_exp and the official timm checkpoint mapping. The weights in this repository come from the V4-Flash-Vision-Exp release, pinned to revision 6821d6ad3681a4b137b066b76094fa82ebd0a380.

Model details

Backbone: 32 layers, hidden size 1024, 16 attention heads, MLP intermediate size 2816.
Patch size: 14 × 14; SwiGLU MLP, RMSNorm and axial 2D RoPE.
Backbone parameters: 411,842,560; stored in BF16.
Default input: 392 × 392 RGB. Rectangular inputs with dimensions divisible by 14 are supported.
forward_features() returns final-normalized NHWC patch features.
forward() returns average-pooled, affine-free RMS-normalized 1024-dimensional embeddings.
The original aligner groups 3 × 3 patches and projects to 4096 dimensions. It is retained only in the separately provided original-format checkpoint.

Usage

pip install 'timm==1.0.30' torch torchvision safetensors pillow

import torch
import timm
from PIL import Image

model = timm.create_model('hf-hub:gatilin/deepseek_vit_412m.deepseek_v4_flash_vision_exp', pretrained=True).eval()
config = timm.data.resolve_model_data_config(model)
transform = timm.data.create_transform(**config, is_training=False)
x = transform(Image.open('image.jpg').convert('RGB')).unsqueeze(0)

with torch.inference_mode():
    embedding = model(x)                  # [1, 1024]
    features = model.forward_features(x)  # [1, 28, 28, 1024]

The default FP32 model loads the BF16 checkpoint into FP32 parameters. For BF16 inference, explicitly move both model and input to a device that supports the required operations and the same dtype.

Files

model.safetensors: native timm classifier-wrapper checkpoint, without aligner or classification head (~824 MB).
config.json: timm Hub configuration.
vision_and_aligner.safetensors: original vision.* and aligner.* keys (~933 MB); this is not a second timm checkpoint.
vision.py, image_processor.py, source_config.json, LICENSE: original source files from the pinned upstream revision.
extract_vit.py: reproducible extraction with validated HTTP range requests; downloads only visual tensors from source shard 1.
validate_vision_exp.py: strict-loading and numerical-equivalence check against the original ViT.
smoke_vision_exp.py: default-resolution 392 × 392 forward check.
source_tensor_index.json: original visual tensor shapes, dtypes and source byte offsets; only these non-contiguous visual ranges are downloaded.
source_inspection.json: source repository inspection record.
extraction.json: provenance, parameter count and SHA256 checksums.
validation.json: validation results and library versions.

To load original-format weights, instantiate ViT and Aligner from vision.py using the source configuration; load the corresponding keys after removing vision. or aligner.. These modules consume patch tensors and patch-grid dimensions rather than raw image batches.

Conversion and validation

The conversion renames attention/MLP keys, replaces the vision. prefix with encoder., and reshapes the linear patch projection from [1024, 588] to [1024, 3, 14, 14]. Tensor data bytes and BF16 precision are preserved.

Strict key/shape loading passed for both backbone and aligner with timm 1.0.30 and torch 2.14.1. FP32 comparisons against the original implementation covered 28 × 28, 28 × 42, 126 × 126 and 126 × 140 inputs, including rectangular grids and incomplete 3 × 3 aligner groups. Maximum absolute errors across all cases were 1.04e-06 for backbone features, 9.83e-07 for aligner tokens, and 1.34e-05 for pooled embeddings. All comparisons passed torch.testing.assert_close(rtol=3e-4, atol=3e-4). A default-resolution 392 × 392 FP32 forward check also passed: features [1,28,28,1024], embedding [1,1024], all values finite. See validation.json for individual results. These conversion checks are not downstream accuracy benchmarks.

Preprocessing and limitations

RGB normalization uses mean and standard deviation (0.5, 0.5, 0.5). The timm evaluation transform uses bicubic resize, border padding, and a fixed canvas. The upstream processor uses an adaptive canvas with gray 127 padding (minimum pixels 147456, maximum image-token budget 384, and maximum width/height ratio 8); timm uses gray 128. Use the included upstream processor when reproducing the original preprocessing policy and image-token ordering, including its N-layout permutation and padding. The extracted ViT and aligner do not include the LLM image-span embeddings or token-layout assembly.

This is an image feature model. A newly added classification head requires training. The independent backbone does not provide the language-model capabilities of the source multimodal model.

中文说明

本仓库从 DeepSeek-V4-Flash-Vision-Exp 的固定版本重新提取视觉权重，提供可直接通过 timm 加载的独立 ViT，以及保留原始键名的 ViT + Aligner 权重。没有额外训练，没有语言模型权重。主文件为 BF16、411.84M 参数；转换仅涉及键名映射和 patch projection 形状重排。已通过严格加载与四种尺寸输入的数值一致性验证。默认 timm 预处理与上游动态画布策略存在差异，详见上文。

License

MIT, following the included upstream LICENSE. This is an independently extracted checkpoint; no affiliation with or endorsement by DeepSeek-AI or the timm maintainers is implied.