Kolibri-1-78B-A3B-Full-MoE-1M-ConstantKV-SOTA-GEFERTIGT-IN-DEUTSCHLAND

Aleph Alpha's Kolibri-1, the full 78B mixture of experts, at one million tokens in a constant 0.21 GB per request.
Same weights. No retraining. Every answer checked against the text.

The full mixture of experts: all 384 routed experts of every block are kept (top six plus the shared expert per token, 3.46B of 78.1B parameters active); the 40 sliding-window blocks and every weight are untouched. These are the numbers of this release, not a ceiling: M-Rail learns from every run you make. Your practice runs become your rules, your map answers what it has seen in milliseconds, and the numbers go up with use.

Native Kolibri-1 against this release

	native Kolibri-1 (FP8 cache)	this release
attention memory per request at 1M tokens	10.76 GB	0.208 GB · 52× less
concurrent requests on two H100 (report's Table 31 accounting)	23 at 256k · 5 at 1M	300 at any length · 60× at 1M
decode on one RTX PRO 6000, 131k tokens	8 requests · 392 tok/s	32 requests · 1,347 tok/s · 3.4×
decode on one RTX PRO 6000, 262k tokens	4 requests · 209 tok/s	32 requests · 957 tok/s · 4.6×
RULER at 1M tokens, category mean of the report	63.5 (reported by Aleph Alpha)	90.1 · 13-task mean 93.9
the same 131k context asked again	20.0 s	0.11 s from the map
a new question to a 1M-token document read once	160 s to read it	6.6–11.6 s per question

European state of the art in its class

The weights are Kolibri-1's, unchanged. In Aleph Alpha's own comparison (Kolibri report, Tables 28–29) Kolibri-1 leads the open models of its class; this release adds the million tokens, the memory and the throughput.

benchmark	Kolibri-1 78.1B / 3.46B	Qwen3.6 35B / 3B	Nemotron 3 Super 120B / 12B	Mistral Small 4 119B / 6B
AIME 2026 (EN)	96.0	91.0	90.4	83.1
AIME 2026 (DE)	90.0	84.4	87.5	78.5
GPQA Diamond (EN)	84.3	83.4	78.0	74.7
GPQA Diamond (DE)	81.3	80.6	76.6	72.9
LiveCodeBench v6	85.9	82.5	82.0	71.2
τ³-bench (Banking)	38.1	10.6	15.5	5.7
IFBench (loose prompt)	78.1	66.1	73.7	49.8

What this release adds

Two layers on top of Kolibri-1:

1. A constant attention state. The ten full-attention blocks keep, per key–value head, the first token, the last 1,024 tokens exact and a store of 34,816 far tokens (int3, Hadamard-rotated, 112 bytes per token). Three quarters of the store are the tokens that received the most attention in the band, one quarter is chosen by the request once the prompt has been read. One exact softmax over first token, band and store. The state does not grow with the context.
2. M-Rail. Kolibri learns from its own runs: record patterns (the "clown", not the colour of its nose) are learned from practice runs and kept as readable, editable rules; a check gate tests every answer against the text without any reference; an answer that fails is read again from an 8k excerpt; every answer comes with the records it was read from; a context seen once is answered from the map in 0.11 s.

RULER at one million tokens

RULER, 13 tasks, RULER's metric, one RTX PRO 6000 (96 GB), vLLM 0.29, FP8 weights. 1M: items 0–7 of each task (n = 8), state K16, int3, request share 0.25, linear prefill from 262k. The rules come from practice runs on other items (32k and 131k); the reference answers only score.

1M tokens	single pass	M-Rail (gate + re-read 8k)
niah single 1 / 2 / 3	100 / 100 / 87.5	100 / 100 / 100
niah multikey 1 / 2 / 3	100 / 75.0 / 12.5	100 / 100 / 100
niah multivalue / multiquery	53.1 / 90.6	100 / 100
variable tracking	97.5	100
common words / frequent words	0.0 / 87.5	100 / 95.8
qa 1 / qa 2	62.5 / 25.0	62.5 / 62.5
mean	68.6	93.9

• Unseen items. On items 8–15, which played no part in building the rules or the gate, the rules alone (no single pass, no 1M prefill: the rule finds the evidence on the CPU in 0.05–0.3 s, Kolibri reads 2–8k tokens) give 100 on all eight needle tasks and on frequent words (mean over 13 tasks 79.4).
• The gate. On 324 answers of earlier runs (131k and native 1M) the gate kept 238 answers of rule families, none of them wrong, and sent all 38 wrong ones to the re-read.
• Native Kolibri at 1M (n = 4, same items): multikey 3: 25, multivalue: 93.8, common words: 0.
• 131k (n = 4, reference engine, RULER's protocol): native 90.0, constant state 90.3.

Throughput and serving

serving	native FP8 cache	constant state K16
memory per request, 256k / 1M	2.7 GB / 10.8 GB	0.21 GB / 0.21 GB
concurrent requests on 2× H100 (report's Table 31 accounting), 256k / 1M	23 / 5	300 / 300
decode at 131k (requests, tokens/s)	8, 392	32, 1,347
decode at 262k (requests, tokens/s)	4, 209	32, 957

131k with the Triton kernels for band and int3 store; 262k with the first complete version of the backend (band and admission in Torch), which decodes at the same rate at 32k, 131k and 262k.

Read once, ask again

Documents read once (/v1/documents, one RTX PRO 6000): a document of 129k tokens is read in 22–28 s, each question to it then takes 3.8–4.1 s (a fresh read of document and question: 17.8 s); a document of 1M tokens is read in 160 s, each question 6.6–11.6 s. The question continues the document's saved state (exact archive in host memory) and reads only its own tokens; the answers carry verdict and evidence like any other. A run on a million-token context is kept in the map as a track of 143 bytes.

Quickstart

hf download Aleph-Alpha/Kolibri-1 --local-dir kolibri
hf download tfwnotops/Kolibri-1-78B-A3B-Full-MoE-1M-ConstantKV-SOTA-GEFERTIGT-IN-DEUTSCHLAND --local-dir k1v
pip install "./k1v[serve]" aleph-alpha-inference
cd k1v && kolibri serve --config kolibri.yaml --set model=../kolibri      # HTTP on :8000

# a document read once, asked many times; each answer reads only the question
curl -s localhost:8000/v1/documents -d '{"text": "...a million tokens..."}'          # -> {"id": "4f1c..."}
curl -s localhost:8000/v1/documents/4f1c.../ask -d '{"question": "What is the special magic number for ...?"}'

Every response carries kolibri: verdict (checked: the rule confirms the answer · copied: every part stands in the text · failed / uncopied: the rule's value is missing / a part of the answer is not in the text, and the answer is replaced by the re-read when the prompt is longer than the excerpt), family, evidence (the records, as token spans and text), from_map, reread. POST /v1/chat/completions speaks the OpenAI format; usage.prompt_tokens_details.cached_tokens shows how much of a document ask came from the session.

kolibri vllm --config kolibri.yaml prints the same configuration as a vllm serve command line (sessions through vllm_xargs: kolibri_save, kolibri_resume).

One file for every switch

kolibri.yaml (all keys optional; the defaults are the configuration of the numbers above):

key	what it does
mode	native (Kolibri's FP8 cache) or state (constant state)
context	longest prompt + answer; rows, RoPE cache and linear prefill follow from it
state.store	far tokens per KV head: K16 (34,816) or K64 (147,456)
state.admission	band admission by attention mass or by the spitze (sharpest single look)
answer.rules	rules learned from Kolibri's own practice runs (kolibri rules learn)
answer.check, answer.reread, answer.budget	the check gate, the re-read and its excerpt size
map.path	compute once: every answer recorded, the same context answered from the map
sessions	documents read once (exact archive in host memory: the FP8 keys and values of the ten full-attention blocks, 10 KB per token, 11 GB per million tokens)
team.size	> 1: independent solutions, majority of their final answers, a judge on a tie

Rules you can read

kolibri rules show --rules rules/practice-131k-32k.rules.json
# niah_multikey_3 {'left': ' special magic uuids for', 'mid': ' is:', 'right': '.\nOne', 'support': [...], 'contra': [...]}
# cwe   {'count': {'left': '.', 'right': ' ', 'k': 9, ...}}
# fwe   {'words': {'k': 3, ...}}
kolibri rules learn --map my-runs.mrail --out my.rules.json      # from your own runs (map.store: full)

A rule is learned from where Kolibri copied its own answers in practice runs, kept with the runs that support and contradict it, and changed by editing a JSON file, without retraining. One record rule serves six needle families:

How it works

• Band and store. The last 1,024 tokens of each full-attention layer are exact; far tokens live in the store, read at their int3 logits. A token leaving the band is admitted by the attention it received there.
• The request's own look. When the prompt has been read, a quarter of the store is chosen again by the sharpest look of the last 64 prompt rows (spans of 16 tokens, one ranking over all heads and layers), read from the exact keys in host memory. The same look ranks the re-read candidates.
• Check gate. The rule of the recognised family binds the records the question asks for (the longest match of the question's values; a key fragment shared by other keys binds nothing); the answer has to name the value of each, compared as text. Counting families: the k most frequent values or words of the text.
• Re-read. Head, the rule's evidence, the best spans of the look, tail: a short native read where Kolibri copies reliably. Its cost does not depend on the context length.
• Compute once. The state is a function of the tokens; the .mrail map (mrail on PyPI) stores each run as a track: the context by its hash, the answer copy-coded against it, 143 bytes for a 1M-token run.

Paper

Building Europe's Frontier AI: Kolibri-1, the Full 78B Mixture-of-Experts Language Model, David Tom Foss, October 2026, doi:10.13140/RG.2.2.30326.77125

• paper/Foss_2026_Kolibri_1M.pdf: the paper, 42 pages
• paper/Kolibri_Figures.pdf: all twelve figures as vector pages
• paper/: LaTeX and TikZ sources (bash paper/build.sh --figures, needs the CTAN font package geist-font), the plotted values in paper/FIGURE_DATA.json

Open for collaboration and questions on any part of this work: d.foss@ieee.org · github.com/DT-Foss · huggingface.co/davidfoss

License

The code in this repository (kolibri_mrail, kolibri_state, mrail) is licensed under the PolyForm Noncommercial License 1.0.0; commercial use needs a license from the author. The Kolibri-1 weights are Aleph Alpha's, under Apache 2.0, and are not part of this repository (see NOTICE). The paper, its sources in paper/ and the figures in assets/ are licensed under CC BY 4.0; notices for the figure-source examples are in paper/THIRD_PARTY_NOTICES.md.

Citation

@misc{foss2026kolibri,
  author = {Foss, David Tom},
  title  = {Building Europe's Frontier AI: Kolibri-1, the Full 78B Mixture-of-Experts Language Model},
  year   = {2026},
  month  = oct,
  doi    = {10.13140/RG.2.2.30326.77125},
  note   = {Preprint}
}

The map format: David Tom Foss, Compute Once: Constant-State Language Models Make Answers Reusable and Agents Cheap, 2026, doi:10.13140/RG.2.2.18140.35202.