tinystories-50m

A 56,902,144-parameter transformer language model trained from scratch on TinyStories, a corpus of simple, repetitive children's stories. It is the 50M scale-up in the tinystories-24m → tinystories-50m lineage.

v2 (2026-09-25): retrained with a larger 12288-vocab BPE tokenizer (was 8192). The 8192-vocab v1 is fully superseded — same repo, same loader, better weights. v1's held-out val loss was 1.6566; v2's is 1.3837.

It writes fluent, on-domain children's stories. It is not a general language model — out-of-domain generation degrades, and it should not be used for anything beyond the story domain it was trained on.

Architecture

Field	Value
Parameters	56,902,144 (exact; verified against the safetensors header)
Layers (L)	16
d_model (D)	512
Heads (H)	8 (head dim 64)
FFN dim	2048 (4× D)
Vocab	12288 (BPE)
Max seq len	512
Embeddings	weight-tied (lm_head = tok)
Norm	RMSNorm (pre-norm, 2 per block + final)
Activation	GELU
Attention	causal, no bias in linear layers
Dtype	float32

Parameter breakdown (sums exactly to 56,902,144):

• token embedding: 12288 × 512 = 6,291,456
• position embedding: 512 × 512 = 262,144
• 16 blocks × 3,146,752 = 50,348,032
  • 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
• final RMSNorm: 512

Training

• Data: TinyStories (ronendagan/TinyStories), 523,389,481 tokens after BPE-12288 re-tokenization (2,119,489 stories, ~9.19 tokens/param), with a 2M-token held-out tail for validation.
• Optimizer: AdamW, cosine LR decay with warmup (peak 6e-4), grad clip 1.0.
• Batch: 64, seq 512 → 32,768 tokens/step.
• Steps: 15,910 (one full epoch). Best checkpoint at step 13,500.
• Hardware: single NVIDIA RTX 5090 (32 GB).
• Final val loss: 1.3924; best val loss 1.3837 (step 13,500). The shipped weights are the end-of-run checkpoint (val 1.3924), within 0.009 of the best.

Evaluated numbers

• Held-out perplexity (TinyStories val split): exp(1.3837) ≈ 3.99 (best ckpt). This is the honest primary metric for a narrow-domain model.

• General zero-shot log-likelihood accuracy (the 12288-vocab tokenizer can read these datasets, so we report them — v1's 8192-vocab tokenizer could not):

  Task	Accuracy	n
  BLiMP	64.00%	200
  ARC-Easy	51.09%	599
  PIQA	45.50%	200
  HellaSwag	54.83%	600

  These are single-shot, zero-shot, no-few-shot, on a 57M model trained on one narrow domain — treat them as a scale reference, not a competitive result.

• Coherence: seeded generations are fluent, on-domain, with consistent characters and correct punctuation. Minor artifacts expected at this scale (occasional garbled quote char, a couple of logical slips).

Files

File	What
model.safetensors	weights (227 MB, 99 tensors, float32)
tokenizer.json	BPE-12288 tokenizer (tokenizers format)
config.json	architecture config
load_model.py	self-contained loader + TinyStoriesGPT class

Usage

from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

What it is and is not

• Is: a small, from-scratch, on-domain story generator. Good for studying how a ~57M transformer learns a narrow, repetitive domain.
• Is not: a general-purpose LM. Do not expect coherent output on code, math, or open-domain text. The low perplexity is domain-specific.