GenFirst

Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling.

• Paper: arXiv:2608.29335
• Code: github.com/ByteDance-Seed/GenFirst
• Project page: tom-zgt.github.io/GenFirst

These weights are released under CC BY-NC 4.0: non-commercial research use only, which is stricter than the MIT licence on the code. See Licence and third-party terms before use.

What this is

GenFirst trains a latent VAE jointly with a generative prior without the latent collapse that usually makes such end-to-end training unstable. The method is a two-phase schedule over the prior and entropy weights: phase 1 lets the prior shape the latent geometry, phase 2 drops both weights tenfold so the decoder recovers detail. The result is a latent space that a diffusion prior fits well while remaining a good autoencoder.

This release covers the EiT (diffusion-prior) instantiation. The autoregressive EAR instantiation from the paper is not released.

Two kinds of artifact are published, and they are used together:

1. End-to-end VAEs — the encoder/decoder produced by the end-to-end stage, each shipped with its latent normalization statistics.
2. Generative priors — SiT models for class-conditional generation, and MMDiT models for text-to-image, each trained on a frozen VAE from (1).

Model details

Developed by	Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang
Model type	Latent diffusion (flow matching / stochastic interpolant) with a jointly trained VAE
Prior architecture	SiT-XL and MMDiT-XL: 28 blocks, width 1152, 16 attention heads (~675M parameters)
Text encoder (text-to-image)	CLIP-L, frozen, not modified by us
Precision	Weights released in fp32; sampling tested in bf16
Framework	PyTorch
Inference code	Required — see Usage. These are raw .pt checkpoints, not diffusers pipelines.

Class-conditional priors

SiT-XL on a frozen end-to-end VAE. Conditioning is a single integer class label in [0, 999] following the standard 1000-class label ordering; there is no text input. Epoch counts are prior-only training epochs.

Checkpoint	Resolution	VAE	gFID	IS	gFID (guided)	IS (guided)	Download
EiT, 80 epochs	256x256	f16d32	2.79	175.4	1.38	271.3	checkpoint · weak-20ep
EiT, 480 epochs	256x256	f16d32	1.60	216.8	0.988	270.7	checkpoint · weak-20ep
EiT, 800 epochs	256x256	f16d32	1.45	226.7	0.969	283.4	checkpoint · weak-20ep
EiT, 80 epochs	512x512	f16d32	–	–	1.63	278.9	checkpoint
EiT, 400 epochs	512x512	f16d32	–	–	1.23	286.1	checkpoint

The 512 models reuse the 256 f16d32 VAE unchanged and frozen; only the prior is trained at 512.

Text-to-image priors

MMDiT-XL/2 with a frozen CLIP-L text encoder, on a frozen end-to-end FLUX VAE, at 512x512. Scores are overall GenEval and DPG-Bench, with no prompt rewriting. The two variants differ only in which VAE they were trained on.

Checkpoint	Resolution	GenEval	DPG-Bench	Download
EiT T2I, variant A	512x512	0.8987	82.60	checkpoint · VAE
EiT T2I, variant B	512x512	0.8920	82.06	checkpoint · VAE

Shared visual latents (generation + understanding)

MMDiT-XL/2 on a frozen Qwen3-VL ViT autoencoder (--vae-arch qwenvit) shaped end-to-end with a generative prior plus representation supervision (SigLIP or VLM-NLL, Table 9 in the paper).

Checkpoint	VAE bundle	Supervision	Resolution	rFID ↓	PSNR ↑	GenEval ↑	DPG-Bench ↑	Linear Probe Top-1 (%) ↑
shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt	vae/qwenvit-siglip-b256-s2/	E2E + SigLIP	512x512	0.39	27.64	0.90	83.76	79.25
shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt	vae/qwenvit-nlltrunk-s2/	E2E + VLM-NLL	512x512	0.46	27.19	0.89	83.49	81.69

Unified text-image modeling (JointMMDiT-XL/2)

Bidirectional block-causal JointMMDiT-XL/2 trained on the jointly E2E-shaped text VAE (Cola) and image VAE (FLUX.1) (vae/e2e-joint-gan-l2-330k/, Table 10 in the paper). The phase-3 SFT checkpoint (-512-sft.pt) is used for text-to-image (GenEval / DPG-Bench), and the phase-2 pre-SFT checkpoint (-512.pt) is used for image-to-text COCO captioning.

Checkpoints	Setting	GenEval ↑	DPG-Bench ↑	CIDEr ↑	BLEU-4 ↑	METEOR ↑	ROUGE-L ↑	SPICE ↑	CLIPScore ↑
eit-joint-e2e-text-e2e-img-512-sft.pt (T2I) & eit-joint-e2e-text-e2e-img-512.pt (I2T)	E2E-joint text + image VAEs (vae/e2e-joint-gan-l2-330k/)	86.57	72.68	33.28	7.76	17.74	34.16	11.81	57.97

End-to-end VAEs

Reconstruction quality of the posterior mean, measured at the resolution the VAE is used at.

VAE	Downsample / channels	Initialized from	rFID	PSNR	SSIM	LPIPS	Download
f8d4	8x, 4ch	SD-VAE (stabilityai/sd-vae-ft-ema)	0.57	25.13	0.71	0.14	Hugging Face
f16d32 (decoder-ft, @256)	16x, 32ch	VA-VAE (LightningDiT)	0.36	25.82	0.74	0.12	Hugging Face
f16d32 (no dec-ft, @512)	16x, 32ch	VA-VAE (LightningDiT)	–	–	–	–	Hugging Face
f8d16 variant A	8x, 16ch	FLUX.1 VAE	0.23	29.43	0.85	0.07	Hugging Face
f8d16 variant B	8x, 16ch	FLUX.1 VAE	0.21	29.37	0.85	0.07	Hugging Face
qwenvit-siglip-b256-s2	8x, 16ch	Qwen3-VL ViT + ViT decoder (SigLIP)	0.39	27.64	–	–	Hugging Face
qwenvit-nlltrunk-s2	8x, 16ch	Qwen3-VL ViT + ViT decoder (VLM-NLL)	0.46	27.19	–	–	Hugging Face
e2e-joint-gan-l2-330k	8x, 16ch (image) + 16d (text)	FLUX.1 VAE + Cola Text VAE	–	–	–	–	Hugging Face
f8d32 (e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k, @512)	8x, 32ch	FLUX.2 VAE	0.041	36.46	0.953	0.026	Hugging Face

A VAE and its latent statistics must be kept together. The prior was trained on latents normalized by those exact statistics; pairing a VAE with the wrong statistics file produces noise, not a degraded image. Each VAE directory is laid out as:

<name>/
  <name>.pt                     VAE weights
  <name>.safetensors            VAE weights in diffusers safetensors format (for FLUX.2 release)
  <name>-latents-stats.pt       latent normalization (scale + bias)
  <name>-discriminator-ckpt.pt  discriminator, only needed to resume end-to-end training
  config.json / vae_params.json diffusers config and explicit scaling_factor / shift_factor (for FLUX.2 release)

For vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k, the per-channel scaling_factor and shift_factor parameters are:

--vae_pth=vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors
--model.vae.scaling_factor=[0.887221,0.886926,0.887096,0.881914,0.886901,0.887307,0.886685,0.886833,0.886540,0.886646,0.887031,0.887073,0.887381,0.886768,0.887299,0.885674,0.886466,0.886795,0.886979,0.886519,0.886949,0.886949,0.887036,0.886828,0.887344,0.887004,0.886816,0.886799,0.887354,0.887523,0.887012,0.886643]
--model.vae.shift_factor=[-0.000623,0.000143,0.000282,0.001599,-0.001093,-0.000477,0.000855,-0.000830,-0.000883,0.001076,-0.000417,0.000200,0.000626,-0.000649,-0.000137,0.001453,-0.000816,-0.000307,-0.000290,0.000662,0.000484,0.000993,0.000574,-0.000139,-0.000914,0.000578,-0.000691,-0.000323,0.000619,-0.000606,-0.000372,0.000437]

Usage

These checkpoints need the training repository; there is no diffusers integration and no from_pretrained entry point.

git clone https://github.com/ByteDance-Seed/GenFirst
cd GenFirst
pip install -r requirements.txt          # install torch first

Class-conditional sampling, writing a small batch of images rather than a full evaluation run:

cd Prior_only
python generate.py \
    --model SiT-XL/1 --resolution 256 \
    --ckpt /path/to/prior.pt \
    --vae-arch f16d32 \
    --vae-ckpt /path/to/e2e-f16d32.pt \
    --latents-stats /path/to/e2e-f16d32-latents-stats.pt \
    --num-fid-samples 64 --label-sampling random \
    --cfg-scale 1.5 --sample-dir samples

--model must match the VAE: SiT-XL/2 for f8d4, SiT-XL/1 for f16d32. Text-to-image sampling, the guidance settings that reproduce the numbers above, and the full evaluation harness are documented in the repository under docs/.

Evaluation

Reported metrics and their protocols:

• gFID / IS — 50,000 generated samples scored with torch-fidelity against the reference statistics used by MAR, at the stated resolution. Sampling is SDE with 250 steps. Guided rows use autoguidance (pointing away from an early checkpoint of the same run) with CFG on top, except the 512 rows, which use plain CFG.
• rFID / PSNR / SSIM / LPIPS — posterior-mean reconstructions over a 50,000-image validation split, following latent-diffusion. rFID here is measured against those validation images, not a training-set reference, so it is not comparable to numbers computed the other way.
• GenEval / DPG-Bench — overall scores, Euler sampling with 50 steps and CFG 7.0 on the interval [0.35, 1.0], no prompt rewriting.

Numbers were produced with the versions pinned in environment.yml. FID is sensitive to reference statistics and to the resizing path, so comparisons against other work are only meaningful under the same protocol.

Training

Training proceeds in two stages. The end-to-end stage trains the VAE jointly with a prior under the two-phase GenFirst schedule (global batch 256; 400k steps at the high weight, then 200k at the low one). The prior-only stage then freezes the VAE, caches its latents, and trains from scratch the prior that is actually evaluated (4M steps at 256, 2M at 512). An optional 10k-step decoder-only finetune between the stages improves unguided gFID by roughly 0.2. Text-to-image follows the same structure, with an MMDiT-L/2 prior shaping the latent during the end-to-end stage and the released MMDiT-XL/2 trained on the frozen result.

Training data consists of publicly available datasets. Individual corpora are not enumerated here; each is subject to its own licence and terms of use, which govern independently of the licence on these weights.

Intended use

Intended for research: reproducing the paper, studying end-to-end VAE and prior training, latent-space analysis, and use as a baseline or initialization for further non-commercial research.

Out of scope. Any commercial use. Generating content about real, identifiable people. Generating sexual, violent, hateful, or otherwise harmful imagery. Any use presenting outputs as real photographs or as factual records. Any safety-critical or decision-making pipeline. These models produce synthetic images with no factual grounding and must not be treated as a source of truth.

No safety filter, watermarking, or content classifier is included. Anyone deploying these weights in a setting where outputs reach other people is responsible for adding those controls.

Limitations and biases

• The class-conditional models generate from a fixed 1000-class label set and cannot follow text. Their outputs reflect the visual distribution and the category taxonomy they were fit to, including the geographic and cultural skew documented for that style of curated image collection.
• The text-to-image models inherit the biases of both their training data and the frozen CLIP-L text encoder. Expect uneven representation across demographics, professions, and cultures, and expect prompt-following to degrade on long, compositional, or counting-heavy prompts. GenEval and DPG-Bench scores measure prompt alignment on those benchmarks only and do not certify safety or fairness.
• Text rendering, hands, and fine structure remain unreliable, as with comparable models at this scale.
• Metrics reported here are distribution-level. A good FID says nothing about the quality or appropriateness of any single sample.
• The 512 class-conditional models are reported with guidance only; their unguided behaviour is untuned.
• A VAE paired with mismatched latent statistics fails silently in the sense that it produces output rather than an error. Check the pairing.

Licence and third-party terms

Weights: CC BY-NC 4.0 — non-commercial research use only, with attribution. This is stricter than the MIT licence on the code and is the binding constraint on these artifacts.

The code in the GitHub repository is MIT, but the repository as a whole is not usable commercially as it stands: the data preprocessing derives from NVIDIA EDM2 under CC BY-NC-SA 4.0, and several vendored encoders and samplers trace back to Meta projects under CC BY-NC 4.0. The repository's NOTICE lists this component by component.

These weights are also derivative of upstream models, each carrying its own terms that continue to apply and that you should verify for your use:

Component	Upstream
f8d4 VAE initialization	stabilityai/sd-vae-ft-ema
f16d32 VAE initialization	VA-VAE, from LightningDiT
f8d16 VAE initialization	FLUX.1 VAE, black-forest-labs
Text encoder	OpenAI CLIP
Prior architecture lineage	REPA, REPA-E, and DiT/SiT before them

Training datasets carry their own terms independently of this licence, and those terms may further restrict what you may do with a model trained on them.

Security

A .pt checkpoint is a pickle: loading one executes arbitrary Python. These checkpoints store an args namespace alongside the tensors, so they cannot be loaded with weights_only=True and require weights_only=False on torch 2.6 and newer. Load checkpoints only from sources you trust, and verify you are downloading from the official repository.

To report a security vulnerability, follow SECURITY.md in the code repository: contact the ByteDance security team rather than opening a public issue.

Citation

@article{zheng2026genfirst,
  title={GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling},
  author={Zheng, Guangting and Zhang, Yiyuan and Yang, Tao and Chen, Yunpeng and Zhu, Rui and Deng, Jiajun and Zhang, Yanyong},
  journal={arXiv preprint arXiv:2608.29335},
  year={2026}
}

Contact

Questions through GitHub Issues or mail to zgt@mail.ustc.edu.cn. Security reports go to the address in SECURITY.md, not to Issues.