rodcasnog/tinystories-bpe-1k

TinyStories BPE tokenizer

The tokenizer of the TinyStories models of Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks: a BPE tokenizer for roneneldan/TinyStories, with 1,023 tokens (<unk> included) split by a GPT-2-style regex. It is not byte-level, so unseen characters map to <unk>. A story ends with <|endoftext|> (id 1023) and is padded with <pad> (id 1024).

Usage

scripts/prep_tinystories.py and configs/tinystories.yaml of binary-discrete-diffusion load it by its id:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("rodcasnog/tinystories-bpe-1k")

License

Apache-2.0.

Citation

@misc{casadonoguerales2026binarydiffusion,
  title         = {Binary Discrete Diffusion: Candidate-Set Diffusion for
                   Logic and Planning Tasks},
  author        = {Casado Noguerales, Rodrigo and von R{\"u}tte, Dimitri
                   and Sch{\"o}lkopf, Bernhard and Hofmann, Thomas
                   and Raoufi, Aran},
  year          = {2026},
  eprint        = {2610.28543},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.28543},
}