BeeNara 🐝 v0.4

Put a document in the right folder, or admit that none fits.

BeeNara reads a document together with your own list of folder names and returns one of them, or "none fits". Its confidence is calibrated. When it is not sure enough, it tells you to ask the user instead of guessing. It is the category decider of PollySort, a local, bilingual document archivist.

For long folder lists (more than 9 folders) see the larger BeeNara Big, which reads the whole folder tree.

What is new in v0.4

• Sorts on its own more often, and more safely. On the new v0.4 benchmark (1,500 tasks with 3–9 folders each, unseen trees and names), the risk rule sorts 59 % of all tasks automatically, and 0.3 % of those land in a wrong folder. v0.3 sorts 21.5 % of the same tasks and gets 2.5 % of them wrong, above its 1 % target: its threshold did not carry over to the new folder names.
• More accurate on folder trees it has never seen: 91.5 % on the new benchmark, v0.3 85.5 % on the same tasks.
• Confidence that ranks errors better: AUROC for telling right from wrong answers 0.92, v0.3 0.81 (new benchmark).
• "None fits" with a similar neighbour folder is recognized in 94.8 % of the cases (v0.3 81.3 %).
• English as good as German: 91 % in both languages on the new benchmark.
• More and broader training data: 32 document types (12 new everyday ones such as utility bills, receipts, warranties, health insurance, pensions, school letters, recipes and notes), written by two teacher models that check each other's output.

Key points

• Any folder names, no retraining. Folder names are plain text, like "Rechnungen", "Tax 2025" or "Bank statements". You can add a short description to each.
• Knows when nothing fits. It recognizes 96.8 % of documents whose folder is missing from the list (v0.2 benchmark).
• Sorts only when it is sure. Two rules are built in, see below.
• Small and fast on a CPU. It is a 332 MB ONNX file, with no GPU and no PyTorch: about 0.08 s per document on a desktop CPU with 4 threads, about 0.2–0.3 s on a laptop.
• German and English. Documents and folder names can be in either language, or mixed.
• Private by design. It was trained only on synthetic documents, with no user data. It runs fully offline.

What it is good for, and what not

Good for	Not for
Sorting letters, invoices, contracts and statements into a user's folders	Decisions with legal, medical or financial consequences
Choosing between 3 and 9 folders, or "none"	Long lists of dozens or hundreds of folders: use BeeNara Big
Laptops and offline tools where a local LLM is too slow	Languages other than German and English
Pipelines that need an "ask the human" signal	Extracting fields such as amounts or dates

Quickstart

pip install onnxruntime tokenizers numpy huggingface_hub
hf download Kwokou/BeeNara beenara.py --local-dir .

from beenara import BeeNara

bn = BeeNara.from_pretrained("Kwokou/BeeNara")          # downloads onnx/ and tokenizer/, 366 MB

d = bn.decide(
    "Rechnung Nr. 4711\nBetrag: 119,00 EUR, zahlbar bis 30.10.2026.",
    ["Rechnungen", "Verträge", "Arztbriefe"],
    descriptions={"Rechnungen": "Rechnungen von Lieferanten"},   # optional, helps with unusual names
    rule="risk",                                                  # or "conformal" (default, as in v0.2)
)
d.category        # 'Rechnungen', or None when the user should decide
d.ask             # False: sure enough to sort automatically
d.distribution    # calibrated probabilities for every folder and '__none__'

language="en" switches the internal question to English. The document language does not matter for the categories.

The two rules. rule="conformal" (default, unchanged from v0.2) sorts when the conformal prediction set at α = 0.1 holds exactly one folder. rule="risk" sorts only when the calibrated probability of the top folder reaches the risk threshold of its list size; this gives the "at most 1 % wrong" guarantee above. v0.4 has a risk threshold for lists of 3–9 folders. For other list sizes, rule="risk" always asks.

Earlier versions stay available under the tags v0.3 and v0.2:

from huggingface_hub import snapshot_download
bn_v03 = BeeNara(snapshot_download("Kwokou/BeeNara", revision="v0.3", allow_patterns=["onnx/*", "tokenizer/*"]))

Results

All numbers come from the same evaluation pipeline for v0.4 and the earlier versions, on the PyTorch weights unless noted. v0.4 was trained twice with different seeds; this repository ships seed 2, chosen on the separate selection set, not on the benchmarks below. v0.3 and v0.2 have one seed each, so comparisons with them are descriptive.

New benchmark (v0.4): 1,500 tasks from a held-out "world": folder trees, folder names and documents never used for training. Half of the tasks take their 3–9 folders from a folder tree (with nearby folders of the same tree), half from a flat list; 24.5 % have no fitting folder.

Metric	v0.4, seed 2 (shipped)	v0.4, seed 1	v0.3
Accuracy, folders and "none fits"	91.5 %	90.9 %	85.5 %
Calibration error (ECE)	0.029	0.037	0.028
AUROC, right vs. wrong answer	0.92	0.92	0.81
Wrongly said "none fits"	7.8 %	7.8 %	10.3 %
"None fits" when a similar neighbour folder exists (96 tasks)	94.8 %	91.7 %	81.3 %
Sorted automatically, risk rule (all 1,500 tasks)	59.0 %	54.1 %	21.5 %
… of which wrong	0.3 %	0.7 %	2.5 %

The paired bootstrap over both v0.4 seeds puts the accuracy gain over v0.3 at 3.8 to 7.4 points (95 % interval).

v0.2 benchmark: the unchanged benchmark of v0.2 (byte-identical file), 1,200 decision tasks on 600 held-out documents, half of them with folder names the model never saw, 25.8 % with no fitting folder.

Metric	v0.4, seed 2 (shipped)	v0.4, seed 1	v0.3	v0.2
Accuracy, folders and "none fits"	95.2 %	96.1 %	95.9 %	94.1 %
"None fits" recall	96.8 %	95.1 %	93.5 %	96.8 %
"None fits" precision	86.2 %	91.6 %	91.7 %	84.5 %
Wrongly said "none fits"	5.4 %	3.0 %	2.9 %	6.2 %
Calibration error (ECE)	0.021	0.020	0.038	0.039
AUROC, right vs. wrong answer	0.95	0.93	0.93	0.66
Sorted automatically, conformal rule	70 %	73 %	69 %	63 %

The shipped ONNX file scores 95.3 % on this benchmark (PyTorch: 95.2 %) and makes the same choice as the PyTorch weights on all 64 parity-checked tasks. It was calibrated on itself after quantization.

Independent test, 70 documents from a different generator (PollySort's test projects), 8 folders, conformal rule. In C2 the right folder is missing; C3 uses English names in shuffled order.

	C1 German	C2 "none fits"	C3 English	Sorted automatically in C1
Names only	100 %	94.3 %	100 %	100 %, all correct
With descriptions	100 %	98.6 %	95.7 %	100 %, all correct

In C2, 4 of 70 documents were sorted automatically into a wrong folder with names only, 1 of 70 with descriptions (v0.3: 4 and 4).

How it works

BeeNara is a cross-encoder with option markers. The question, every folder name and the document go into one sequence and through one forward pass:

[CLS] question [SEP] [MASK] folder 1 [MASK] folder 2 … [SEP] document start … document end [SEP]

• Each folder is scored at its own [MASK].
• A separate head reads [CLS] plus the shape of the folder scores and gives p("none fits").
• Both combine into one distribution: p(none) = σ(e), p(folder i) = (1 − p(none)) · softmax(s)ᵢ.

Part	Choice
Encoder	mmBERT-base, a multilingual ModernBERT: 22 layers, 768 dims, 256k vocabulary
Head	one Transformer layer, a folder scorer and a "none" scorer
Input	up to 512 tokens; 192 for question and folders; of the document, the start and the last 30 %
Shipped file	ONNX, weights in 8 bit (MatMulNBits, block 32), embeddings in INT8, 332 MB
Calibration	temperature scaling; split-conformal set at α = 0.1; risk threshold per list size (Learn-then-Test, ≤ 1 % wrong among automatic decisions, δ = 0.1), fitted on 4,000 held-out tasks

Training

• Data. Only synthetic documents, no user data:
  • 30,503 documents in the training world after all checks: new documents from two teacher models, the v0.3 documents re-sorted into new folder trees, and rewritten template documents. Every teacher document was checked against the facts it was built from, and the other teacher checked its document type;
  • 28,524 template documents from v0.1/v0.2 with simulated OCR noise;
  • 32 document types, 12 of them new everyday types;
  • real OCR text: pages were rendered, degraded (scan, phone photo, skew) and read with Tesseract;
  • 3,000 folder trees in German, English and mixed, for private households, freelancers, small firms, construction offices, associations and law offices;
  • about 20 % English mail-triage tasks from Open-Jev (CC0).
• Episodes. Every step draws a new task for each document: 3–9 folders from its tree or from the catalogue, in random order, with hard distractors and similar neighbour folders; in 25 % of the tasks the right folder is removed.
• Kept apart by tests. Three disjoint sets of names and a separate held-out world for selection, calibration and the benchmark. Overlap was checked on normalized names, substrings and descriptions; 116 training documents that came too close to held-out ones were removed. No leakage remained.
• Recipe of the shipped model. As in v0.3 without label smoothing. We also trained two variants with two seeds each: weight averaging (SWA) and distillation from BeeNara Big. Neither was more accurate (paired bootstrap over both seeds); SWA exceeded the 1 % target in one seed, and distillation sorted only about half as many documents automatically. Both were dropped.
• Setup: 3 epochs at batch 16, AdamW with encoder at 2.5e-5 and head at 1e-4, 5 % warmup, then cosine; bf16; frozen embeddings; best checkpoint by validation NLL (step 10,500).
• Hardware: one NVIDIA RTX A6000 (48 GB), about 45 minutes per run.

Environmental impact

We did not have a power meter. The numbers are an upper-bound estimate following Lacoste et al. 2019.

	Compute	Energy, at most	CO₂eq, at most
This model (one training run)	about 0.75 h on one GPU	0.2 kWh	0.1 kg
The whole v0.4 project	about 125 GPU hours on two RTX A6000 (teacher models ~52 h, training of all small and Big models ~58 h, calibration and evaluation ~13 h) over 7 days	97 kWh	37 kg

Assumptions: at most 300 W per GPU while in use, plus 350 W for the rest of the machine over the whole 7 days; grid 0.38 kg CO₂eq/kWh (German mix 2023/24). Not included: the pretraining of mmBERT and of the teacher models, and the v0.3 documents that v0.4 re-used (counted in v0.3).

Credits and inspiration

Source	What BeeNara takes from it
mmBERT (paper), JHU CLSP, MIT	the encoder weights
ModernBERT, Warner et al. 2024	the encoder architecture behind mmBERT
laya, convaiinnovations, Apache-2.0	the idea of one [MASK] marker per option, scored in one pass
GLiClass	all labels in one sequence, one forward pass
Guo et al. 2017	temperature scaling
Angelopoulos & Bates 2021	split-conformal prediction
Angelopoulos et al. 2021, Learn then Test	the risk thresholds with a finite-sample guarantee
Xia et al., Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It	the effect that made us drop label smoothing
Qwen3.8-27B, Qwen team, Apache-2.0	wrote training and held-out documents and drafted folder trees; checked the other teacher's names, trees and document types; run locally
gpt-oss-120b, OpenAI, Apache-2.0	wrote training and held-out documents, all folder names, and drafted folder trees; checked the other teacher's trees and document types; run locally
Open-Jev, ZefanCai, CC0-1.0	about 20 % of the training tasks, from mailroom-control-v1
Tesseract OCR, Apache-2.0	the OCR text in training
ONNX Runtime	weight-only 8-bit quantization (MatMulNBits)

Limitations

• Automatic sorting is certified for 3–9 folders only. For other list sizes, rule="risk" always asks. With the conformal rule BeeNara still sorts, without the 1 % guarantee.
• Not for long folder lists. A pre-selection of 9 folders by an embedding model (multilingual-e5-small) followed by BeeNara reached 69–76 % on large trees. Use BeeNara Big for those.
• Says "none fits" a little too often on clean lists. On the v0.2 benchmark the shipped seed wrongly answers "none fits" in 5.4 % of the tasks (v0.3: 2.9 %), in exchange for recognizing more missing folders (96.8 % vs. 93.5 %).
• Weaker cases on the new benchmark: short notes (60 % of 20), receipts and vehicle papers (82 %), OCR text from degraded scans (75 % of 81), and folders without a description (87 %, 97 % with one).
• Synthetic training data. Real scans, letterheads and handwriting are messier. Expect a gap until the model is checked on real, consented documents.
• German and English only.
• BeeNara supports a decision; it does not make one. Use it where a wrong folder can be undone.

Files

File	Content	Size
onnx/beenara.onnx	the model	332 MB
onnx/calibration.json	temperatures, conformal threshold, risk thresholds and input truncation, fitted on this file; bound to it by a SHA-256 fingerprint	< 1 KB
tokenizer/	tokenizer of mmBERT-base	34 MB
beenara.py	inference: input format, calibration, both decision rules	9 KB

License

Apache-2.0, see LICENSE. The weights derive from mmBERT-base, which is MIT-licensed; see NOTICE. Developed by Akinara as part of PollySort.

@misc{beenara2026,
  title  = {BeeNara: a calibrated, abstaining category decider for document sorting},
  author = {Akinara},
  year   = {2026},
  url    = {https://huggingface.co/Kwokou/BeeNara}
}