notyost/claudius-purus-110m-g

Claudius Purus 110M-G

A generative language model with a knowledge cutoff of 0 BCE — the full seven-language variant, including Vedic Sanskrit and the Epic of Gilgamesh. Trained from scratch (architecture, tokenizer, weights) exclusively on text composed before the Common Era. It has never seen a word written after antiquity.

Why NC: two sources — GRETIL's Vedic corpus and the electronic Babylonian Library's Gilgamesh — are CC BY-NC-SA. Whether trained weights inherit a text license is legally unsettled; we honor the licensors' one request regardless, so this model is research/play only. For commercial use, its sibling claudius-purus-110m (clean six-language variant, CC BY-SA 4.0) exists for exactly that purpose.

The facts

• 27.92M training tokens: Ancient Greek, Latin, Akkadian, Sumerian, consonantal Biblical Hebrew, Classical Chinese, Vedic Sanskrit — plus the Standard Babylonian and Old Babylonian Gilgamesh. 996 works spanning ~21 centuries of composition. Every inclusion governed by a public dating manifest (dates, confidence, rationale — and the exclusions: Strabo, Ovid, Livy's Periochae, Śaṅkara's commentary, Qianlong's poems…).
• 110.9M parameters, GPT-2-small architecture (12L/768/12H, 1024 ctx, 32k corpus-trained byte-level BPE), fp16, 16 epochs, seed 1453. Exported with exact (0.00e+00) logit parity to HF format.
• Best val loss 3.664 (ppl 39.0); per-language below. Training cost ~$2 of rented H100. The corpus, not compute, is the binding constraint — there is no more pre-CE text.

Showcase: the ibru experiment

This model is the "after" of a controlled intervention. Its predecessor was identical except for one text: the Epic of Gilgamesh (13k words — 0.05% of the corpus, but ~40% of the model's lifetime exposure to Akkadian ibru, "friend"). Same seed, frozen tokenizer, byte-identical validation set; Latin/Greek control probes held steady. Result: the Akkadian negation cadence re-aimed from contract law ("he shall not sue, he shall not hand over") to mortality ("his strength does not return"; the ghost; the weary; the ancient), and the bare cry ib-rī began answering with first-person memory ("in the palace, he seated me"). One epic bent the word "friend" from the courtroom toward the grave. That is what this model is for: measuring what surviving texts do to meanings.

What it is — and is not

A rhapsode, not an oracle: strong period style and grammar (Ciceronian prose, Homeric formulae, psalm-register Hebrew, Brāhmaṇa ritual prose, cuneiform conventions — including, charmingly, the lacuna brackets of damaged tablets), near-zero factual reliability, no reasoning. It continues text; it does not chat. Biases stated plainly: the corpus is elite, literate, male, and survivorship-filtered — this simulates surviving literature, not ancient populations. Hebrew is unpointed consonantal text. Sanskrit binds weakly (its mass is formulaic ritual prose; expect the altar to swallow your prompt).

Evaluation

lang	val tokens	NLL/token	per-token ppl
Akkadian	110,681	2.499	12.2
Sanskrit	38,341	3.602	36.7
Ancient Greek	89,124	3.765	43.2
Latin	36,951	4.729	113.2
Classical Chinese	45,905	6.056	426.7
ALL	321,002	3.748	42.4

⚠ Per-token perplexity is incomparable across tokenization densities (Akkadian ≈ 4.4 predictable syllable-tokens/word; Chinese ≈ 1 morpheme-token/char); per-word NLL inverts the ranking (Greek is the model's strongest language). Val split contains no Hebrew/Sumerian works (hash-split coverage gap, noted for v2).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("notyost/claudius-purus-110m-g")
model = AutoModelForCausalLM.from_pretrained("notyost/claudius-purus-110m-g")
ids = tok("ib-ri", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=50, do_sample=True,
                     temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

Provenance & thanks

See ATTRIBUTION.md — Perseus, Open Greek & Latin, ORACC, Tanach.us, Kanripo, GRETIL, and the electronic Babylonian Library. Decades of scholarly digitization, honored license and all, made this possible for the price of a sandwich. Ib-ri na-piš-ti — my friend, my life — the model said it before it ever read the tablet where it belongs.