Complete Suite of BPE Tokenizers for TinyStories
This repository contains a full suite of custom BPE tokenizers trained from scratch on 1,000,000 stories (about 200M words) from the roneneldan/TinyStories dataset.
Optimizing vocabulary size is critical when training Small Language Models (SLMs) on resource-constrained hardware like Mac M1 (8GB RAM). This benchmark provides data-driven evidence for choosing the right vocabulary size.
Benchmark Results (Tested on 5,000 Validation Stories — M1 Pro, 2026)
Updated 2026-08-30: re-trained on 1M stories, verified on M1 Pro (previous numbers were M1 Air). Results within 1-3% of original — sweet spot unchanged.
| Vocab Size | Total Tokens | Compression Ratio | Avg Tokens/Story | Min Tokens/Story | Max Tokens/Story | Speed (Tokens/sec) | HW |
|---|---|---|---|---|---|---|---|
| 1024 | 1,279,530 | 3.18 | 255.9 | 64 | 1296 | ~1,079,000 | M1 Pro |
| 1536 | 1,164,651 | 3.49 | 232.9 | 57 | 1193 | ~992,000 | M1 Pro |
| 2048 | 1,106,714 | 3.67 | 221.3 | 53 | 1132 | ~948,000 | M1 Pro |
| 2560 | 1,072,143 | 3.79 | 214.4 | 53 | 1099 | ~916,000 | M1 Pro |
| 3072 | 1,048,720 | 3.88 | 209.7 | 52 | 1058 | ~903,000 | M1 Pro |
| 4096 | 1,017,813 | 3.99 | 203.6 | 51 | 1038 | ~875,000 | M1 Pro |
| 8192 | 978,335 | 4.15 | 195.7 | 49 | 1018 | ~845,000 | M1 Pro |
| 16384 | 970,489 | 4.19 | 194.1 | 49 | 1005 | ~833,000 | M1 Pro |
| 50257 | 968,439 | 4.20 | 193.7 | 49 | 1004 | ~825,000 | M1 Pro |
Previous M1 Air numbers (2025): 1024 2,700,793/3.07/270.1/554K … 50257 1,980,834/4.18/198.1/354K — same trend, ~2× slower on Air.
Key Insights
- The 1536-2048 Sweet Spot: Moving from 1024 to 1536/2048 gives the sharpest increase in compression efficiency (3.18 → 3.67). For lightweight models, this range offers the best trade-off between sequence compression and embedding matrix size.
- The 4096 Diminishing Returns: A vocabulary size of 4096 captures almost all structure needed for TinyStories (compression ratio 3.99).
- The 50257 Overkill: The standard GPT-2 vocabulary (50,257) is massive overkill. Increasing vocabulary size by 12x (compared to 4096) saves a mere 10 tokens per story on average. For TinyStories, larger vocabularies just bloat the model's weights without giving any real context-window benefits.
How to use
Since all files are stored in the root directory, you can load any specific tokenizer directly by specifying the file name:
from tokenizers import Tokenizer
# Load the optimal 2048 tokenizer
tokenizer = Tokenizer.from_pretrained(
"morginalium/tinystories-tokenizers",
filename="tokenizer_2048.json"
)
encoded = tokenizer.encode("Once upon a time, there was a little pup.")
print(encoded.ids)