rodcasnog/binary-discrete-diffusion

Binary Discrete Diffusion: text models

The text models of Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks: one diffusion transformer per corpus and noise kernel (masked, uniform, binary_symmetric, binary_absorbing). Each <corpus>/<kernel>/ is a run directory of binary-discrete-diffusion, trained with configs/paper/<corpus>/<kernel>.yaml: the composed config in .hydra/config.yaml and the EMA parameters of the final step in checkpoints_ema/<step>/.

Corpus	Directory	Checkpoint	Tokenizer
Text8	text8/<kernel>	checkpoints_ema/1000000	characters, written by scripts/prep_text8.py
TinyStories	tinystories/<kernel>	checkpoints_ema/500000	rodcasnog/tinystories-bpe-1k
OpenWebText	owt/<kernel>	checkpoints_ema/250000	dvruette/openwebtext-bpe-1k

Usage

From a clone of the code repository, after uv sync:

uv run hf download rodcasnog/binary-discrete-diffusion --include "text8/binary_symmetric/*" --local-dir outputs/hf
uv run python -m binary_diffusion.sample checkpoint_dir=outputs/hf/text8/binary_symmetric/checkpoints_ema/1000000 \
    sampler.nfe=256 sampler.spacing=log sampler.eta=1 sampler.lam=1 out=outputs/text8-bsym/samples

Text8 models read the tokenizer in data/text8, TinyStories samples take their lengths from data/tinystories, and NELBO and MAUVE read the corpus; the code's README lists the data and evaluation commands.

License

Apache-2.0.

Citation

@misc{casadonoguerales2026binarydiffusion,
  title         = {Binary Discrete Diffusion: Candidate-Set Diffusion for
                   Logic and Planning Tasks},
  author        = {Casado Noguerales, Rodrigo and von R{\"u}tte, Dimitri
                   and Sch{\"o}lkopf, Bernhard and Hofmann, Thomas
                   and Raoufi, Aran},
  year          = {2026},
  eprint        = {2610.28543},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.28543},
}