Hugging Face's logo

baobabtech
/
millid-tiny-v1.5

Text Classification
PyTorch
ONNX
Safetensors
multilingual
language-identification
byte-level
Eval Results (legacy)
🇪🇺 Region: EU
Model card Files
xet
 Community
5
byamasupatrick commited on
Commit
3057a7f
·
verified ·
1 Parent(s): beaf38d

Model card for the public release

Browse files
Files changed (1) hide show
  1. README.md +366 -361
README.md CHANGED
@@ -1,361 +1,366 @@
1
- ---
2
- library_name: pytorch
3
- pipeline_tag: text-classification
4
- tags:
5
- - language-identification
6
- - onnx
7
- datasets:
8
- - baobabtech/millid-dataset
9
- ---
10
-
11
- # MillID tiny: MultiLingual Language IDentifier
12
-
13
- A byte-level, encoder-only transformer that names the language of a passage: **1641 language-script
14
- labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 2.9M parameters, trained from
15
- scratch on [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset).
16
-
17
- ## Use it
18
-
19
- No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or
20
- JavaScript, and the two give the same answers (one encoder, ported line for line, both tested
21
- against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)).
22
-
23
- ### Python
24
-
25
- ```bash
26
- pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git" # [gpu] for CUDA; the repo is private
27
- ```
28
-
29
- ```python
30
- from millid import MillID
31
-
32
- model = MillID.from_pretrained("tiny-v1.5") # or "baobabtech/millid-tiny-v1.5"; a private repo needs HF_TOKEN
33
- model.predict("Lave men ou ak savon anvan ou manje.", k=3)
34
- # [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
35
- model.predict(["one text", "another"], k=3) # one list of answers per text
36
- model.identify_document(long_text) # any length: its languages and segments
37
- ```
38
-
39
- ```bash
40
- millid predict "Lave men ou ak savon anvan ou manje." --model tiny-v1.5 -k 3
41
- ```
42
-
43
- ### JavaScript: Node and browsers
44
-
45
- From a clone of [baobab-tech/millid](https://github.com/baobab-tech/millid) (`js/`, not on npm yet):
46
-
47
- ```bash
48
- npm install ./millid/js onnxruntime-node # onnxruntime-web for a browser app
49
- ```
50
-
51
- ```js
52
- import { MillID } from "@baobabtech/millid"; // Node
53
- const model = await MillID.fromPretrained("tiny-v1.5"); // HF_TOKEN or a saved login for a private repo
54
- await model.predict("Lave men ou ak savon anvan ou manje.", 3);
55
- await model.identifyDocument(longText);
56
- ```
57
-
58
- In a browser, `import { MillID } from "@baobabtech/millid/web"` and pass onnxruntime-web:
59
- `MillID.fromPretrained("tiny-v1.5", { ort })`. 11 MB: small enough to load in a web page.
60
-
61
- ### PyTorch
62
-
63
- `millid.py` in this repository is the training code. It reads `model.safetensors`, to fine-tune or
64
- export:
65
-
66
- ```python
67
- from millid import MillID # millid.py from this repository, not the pip package
68
-
69
- model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
70
- ```
71
-
72
- ### The ONNX graph
73
-
74
- For each UTF-8 byte of a passage (NFKC, whitespace collapsed, cut to 255 bytes) after a CLS token:
75
- `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and
76
- `mask` (bool, `[batch, bytes]`). It returns `logits` (float32, `[batch, 1641]`); a softmax gives the
77
- probabilities. `millid.encode_batch` (Python) and `encodeBatch` (JavaScript) build the inputs; script
78
- ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
79
-
80
- ## Architecture
81
-
82
- UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character
83
- 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided
84
- convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE,
85
- 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of
86
- its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
87
-
88
- ## Evaluation against GlotLID and fastText
89
-
90
- `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration
91
- of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them.
92
- `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to
93
- MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR
94
- and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is
95
- Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB,
96
- 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as
97
- `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common`
98
- score every model on the passages whose language all of them know: the head-to-head on equal terms.
99
-
100
- | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
101
- |---|---|---|---|---|---|---|---|---|---|
102
- | validation | millid | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 | 1626 |
103
- | test | millid | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | 1621 |
104
- | test | glotlid | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1621 |
105
- | test | glotlid@millid | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1621 |
106
- | test | fasttext | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 1621 |
107
- | test | fasttext@millid | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 1621 |
108
- | test-common | millid | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | 213 |
109
- | test-common | glotlid | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 213 |
110
- | test-common | glotlid@millid | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 213 |
111
- | test-common | fasttext | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 213 |
112
- | test-common | fasttext@millid | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 213 |
113
- | udhr | millid | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | 358 |
114
- | udhr | glotlid | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 358 |
115
- | udhr | glotlid@millid | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 358 |
116
- | udhr | fasttext | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 358 |
117
- | udhr | fasttext@millid | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 358 |
118
- | udhr-common | millid | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | 170 |
119
- | udhr-common | glotlid | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 170 |
120
- | udhr-common | glotlid@millid | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 170 |
121
- | udhr-common | fasttext | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 170 |
122
- | udhr-common | fasttext@millid | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 170 |
123
- | udhr-all | millid | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | 429 |
124
- | udhr-all | glotlid | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 429 |
125
- | udhr-all | glotlid@millid | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 429 |
126
- | udhr-all | fasttext | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 429 |
127
- | udhr-all | fasttext@millid | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 429 |
128
-
129
- Top-1 accuracy by passage length:
130
-
131
- | Set | Model | short | sentence | paragraph |
132
- |---|---|---|---|---|
133
- | validation | millid | 0.9312 | 0.9788 | 0.9850 |
134
- | test | millid | 0.9333 | 0.9797 | 0.9833 |
135
- | test | glotlid | 0.9664 | 0.9853 | 0.9911 |
136
- | test | glotlid@millid | 0.9674 | 0.9858 | 0.9912 |
137
- | test | fasttext | 0.2760 | 0.2501 | 0.1958 |
138
- | test | fasttext@millid | 0.2764 | 0.2506 | 0.1959 |
139
- | test-common | millid | 0.9125 | 0.9685 | 0.9725 |
140
- | test-common | glotlid | 0.9594 | 0.9876 | 0.9947 |
141
- | test-common | glotlid@millid | 0.9609 | 0.9885 | 0.9949 |
142
- | test-common | fasttext | 0.9112 | 0.9562 | 0.9651 |
143
- | test-common | fasttext@millid | 0.9125 | 0.9581 | 0.9655 |
144
-
145
- ### Leaderboard: test
146
-
147
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is
148
- `reports/test/leaderboard.csv`.
149
-
150
- The 20 languages with the most passages:
151
-
152
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
153
- |---|---|---|---|---|---|---|---|
154
- | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.928 | 0.808 | 0.808 |
155
- | `eng_Latn` | English | 1444 | 0.917 | 0.858 | 0.859 | 0.653 | 0.648 |
156
- | `spa_Latn` | Spanish | 1136 | 0.931 | 0.950 | 0.950 | 0.574 | 0.574 |
157
- | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
158
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.785 | 0.513 | 0.512 |
159
- | `por_Latn` | Portuguese | 657 | 0.947 | 0.947 | 0.949 | 0.756 | 0.755 |
160
- | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.993 | 0.922 | 0.922 |
161
- | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.896 | 0.539 | 0.539 |
162
- | `swh_Latn` | Swahili | 592 | 0.848 | 0.918 | 0.916 | 0.346 | 0.346 |
163
- | `mar_Deva` | Marathi | 589 | 0.965 | 0.989 | 0.990 | 0.926 | 0.926 |
164
- | `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.426 | 0.426 |
165
- | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.999 | 0.232 | 0.232 |
166
- | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.985 | 0.861 | 0.860 |
167
- | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.994 | 0.951 | 0.951 |
168
- | `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.949 | 0.462 | 0.462 |
169
- | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.948 | 0.949 | 0.000 | 0.000 |
170
- | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.991 | 0.994 | 0.976 | 0.976 |
171
- | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
172
- | `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.968 | 0.968 | 0.915 | 0.914 |
173
- | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.986 | 0.000 | 0.000 |
174
-
175
- Where MillID gains most over GlotLID:
176
-
177
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
178
- |---|---|---|---|---|---|---|---|
179
- | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
180
- | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
181
- | `ajp_Arab` | South Levantine Arabic | 28 | 0.945 | 0.581 | 0.581 | 0.000 | 0.000 |
182
- | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.621 | 0.000 | 0.000 |
183
- | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.651 | 0.000 | 0.000 |
184
- | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.667 | 0.000 | 0.000 |
185
- | `und_Geor` | und | 32 | 0.877 | 0.694 | 0.694 | 0.000 | 0.000 |
186
- | `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.761 | 0.000 | 0.000 |
187
- | `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.822 | 0.000 | 0.000 |
188
- | `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.786 | 0.000 | 0.000 |
189
- | `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.800 | 0.000 | 0.000 |
190
- | `und_Kana` | und | 136 | 0.982 | 0.862 | 0.862 | 0.000 | 0.000 |
191
- | `und_Yiii` | und | 38 | 1.000 | 0.894 | 0.884 | 0.000 | 0.000 |
192
- | `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.625 | 0.842 | 0.000 | 0.000 |
193
- | `dag_Latn` | Dagbani | 149 | 0.767 | 0.662 | 0.660 | 0.000 | 0.000 |
194
-
195
- Where MillID loses most to GlotLID:
196
-
197
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
198
- |---|---|---|---|---|---|---|---|
199
- | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
200
- | `ttj_Latn` | Tooro | 28 | 0.485 | 0.945 | 0.945 | 0.000 | 0.000 |
201
- | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 1.000 | 0.000 | 0.000 |
202
- | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.823 | 0.825 | 0.007 | 0.007 |
203
- | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
204
- | `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
205
- | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.783 | 0.000 | 0.000 |
206
- | `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.944 | 0.178 | 0.178 |
207
- | `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 1.000 | 0.000 | 0.000 |
208
- | `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.902 | 0.181 | 0.181 |
209
- | `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
210
- | `nyf_Latn` | Giryama | 10 | 0.645 | 0.952 | 0.870 | 0.000 | 0.000 |
211
- | `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.992 | 0.000 | 0.000 |
212
- | `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.824 | 0.000 | 0.000 |
213
- | `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 1.000 | 0.000 | 0.000 |
214
-
215
- ### Leaderboard: udhr
216
-
217
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. The full table, with precision and recall, is
218
- `reports/udhr/leaderboard.csv`.
219
-
220
- The 20 languages with the most passages:
221
-
222
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
223
- |---|---|---|---|---|---|---|---|
224
- | `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.945 | 0.945 |
225
- | `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.436 | 0.436 |
226
- | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.868 | 0.468 | 0.468 |
227
- | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
228
- | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
229
- | `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.854 | 0.000 | 0.000 |
230
- | `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.986 | 0.000 | 0.000 |
231
- | `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 1.000 | 0.876 | 0.876 |
232
- | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
233
- | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.877 | 0.000 | 0.000 |
234
- | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
235
- | `snn_Latn` | Siona | 136 | 0.631 | 0.514 | 0.514 | 0.000 | 0.000 |
236
- | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.680 | 0.000 | 0.000 |
237
- | `lat_Latn` | Latin | 118 | 0.996 | 0.991 | 0.991 | 0.000 | 0.000 |
238
- | `hat_Latn` | Haitian Creole | 117 | 0.991 | 0.979 | 0.979 | 0.699 | 0.699 |
239
- | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 1.000 | 0.806 | 0.806 |
240
- | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 1.000 | 0.877 | 0.877 |
241
- | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
242
- | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.996 | 0.399 | 0.396 |
243
- | `chr_Cher` | Cherokee | 115 | 0.671 | 0.663 | 0.663 | 0.000 | 0.000 |
244
-
245
- Where MillID gains most over GlotLID:
246
-
247
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
248
- |---|---|---|---|---|---|---|---|
249
- | `mic_Latn` | Mi'kmaw | 59 | 0.991 | 0.427 | 0.427 | 0.000 | 0.000 |
250
- | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.167 | 0.000 | 0.000 |
251
- | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.483 | 0.000 | 0.000 |
252
- | `sag_Latn` | Sango | 59 | 0.991 | 0.578 | 0.578 | 0.189 | 0.189 |
253
- | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.500 | 0.000 | 0.000 |
254
- | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.154 | 0.000 | 0.000 |
255
- | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.462 | 0.000 | 0.000 |
256
- | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.731 | 0.000 | 0.000 |
257
- | `tah_Latn` | Tahitian | 57 | 0.973 | 0.813 | 0.813 | 0.966 | 0.966 |
258
- | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.580 | 0.111 | 0.111 |
259
- | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | 0.000 | 0.000 |
260
- | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.739 | 0.000 | 0.000 |
261
- | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 | 0.000 | 0.000 |
262
- | `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.859 | 0.487 | 0.487 |
263
- | `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.098 | 0.000 | 0.000 |
264
-
265
- Where MillID loses most to GlotLID:
266
-
267
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
268
- |---|---|---|---|---|---|---|---|
269
- | `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.990 | 0.000 | 0.000 |
270
- | `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.912 | 0.000 | 0.000 |
271
- | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.434 | 0.015 | 0.015 |
272
- | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.378 | 0.000 | 0.000 |
273
- | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.882 | 0.874 | 0.000 | 0.000 |
274
- | `idu_Latn` | Idoma | 51 | 0.603 | 0.879 | 0.938 | 0.000 | 0.000 |
275
- | `ell_Grek` | Greek | 115 | 0.671 | 1.000 | 1.000 | 1.000 | 1.000 |
276
- | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.906 | 0.000 | 0.000 |
277
- | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.974 | 0.000 | 0.000 |
278
- | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.459 | 0.459 | 0.000 | 0.000 |
279
- | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.957 | 0.975 | 0.975 |
280
- | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.727 | 0.474 | 0.474 |
281
- | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 1.000 | 0.389 | 0.389 |
282
- | `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.963 | 0.000 | 0.000 |
283
- | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
284
-
285
- ### Leaderboard: test-common
286
-
287
- MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. The full table, with precision and recall, is
288
- `reports/test-common/leaderboard.csv`.
289
-
290
- The 20 languages with the most passages:
291
-
292
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
293
- |---|---|---|---|---|---|---|---|
294
- | `fra_Latn` | French | 1652 | 0.943 | 0.955 | 0.955 | 0.963 | 0.963 |
295
- | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
296
- | `spa_Latn` | Spanish | 1136 | 0.937 | 0.958 | 0.958 | 0.941 | 0.941 |
297
- | `tam_Taml` | Tamil | 754 | 0.997 | 1.000 | 1.000 | 0.997 | 0.997 |
298
- | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.886 | 0.957 | 0.954 |
299
- | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
300
- | `kan_Knda` | Kannada | 601 | 0.973 | 0.994 | 0.994 | 0.999 | 0.999 |
301
- | `zul_Latn` | Zulu | 594 | 0.883 | 0.904 | 0.904 | 0.970 | 0.970 |
302
- | `swh_Latn` | Swahili | 592 | 0.893 | 0.954 | 0.953 | 0.945 | 0.945 |
303
- | `mar_Deva` | Marathi | 589 | 0.968 | 0.989 | 0.990 | 0.987 | 0.987 |
304
- | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
305
- | `hun_Latn` | Hungarian | 576 | 0.989 | 1.000 | 1.000 | 0.980 | 0.980 |
306
- | `afr_Latn` | Afrikaans | 569 | 0.972 | 0.985 | 0.985 | 0.986 | 0.986 |
307
- | `tel_Telu` | Telugu | 555 | 0.997 | 0.998 | 0.998 | 0.999 | 0.999 |
308
- | `rus_Cyrl` | Russian | 528 | 0.945 | 0.985 | 0.985 | 0.927 | 0.927 |
309
- | `ory_Orya` | Odia (individual language) | 511 | 0.994 | 0.997 | 1.000 | 0.996 | 0.996 |
310
- | `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.977 | 0.978 | 0.856 | 0.856 |
311
- | `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.970 | 0.970 | 0.967 | 0.967 |
312
- | `asm_Beng` | Assamese | 467 | 0.997 | 0.999 | 0.999 | 0.997 | 0.997 |
313
- | `ben_Beng` | Bangla | 466 | 0.988 | 0.997 | 0.997 | 0.989 | 0.989 |
314
-
315
- Where MillID gains most over GlotLID:
316
-
317
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
318
- |---|---|---|---|---|---|---|---|
319
- | `ewo_Latn` | Ewondo | 134 | 0.959 | 0.879 | 0.883 | 0.860 | 0.860 |
320
- | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
321
- | `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.906 | 0.913 | 0.852 | 0.852 |
322
- | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
323
- | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.877 | 0.892 | 0.859 | 0.859 |
324
- | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.952 | 0.981 | 0.981 |
325
- | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.925 | 0.933 | 0.979 | 0.979 |
326
- | `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.862 | 0.862 |
327
- | `bjn_Arab` | Banjar | 116 | 0.991 | 0.834 | 0.987 | 0.453 | 0.960 |
328
- | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
329
- | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.993 | 0.961 | 0.961 |
330
- | `san_Deva` | Sanskrit | 239 | 0.990 | 0.989 | 0.989 | 0.970 | 0.970 |
331
- | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 1.000 | 0.977 | 0.977 |
332
- | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 1.000 | 0.984 | 0.984 |
333
- | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 1.000 | 0.994 | 0.994 |
334
-
335
- Where MillID loses most to GlotLID:
336
-
337
- | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
338
- |---|---|---|---|---|---|---|---|
339
- | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.857 | 0.860 | 0.029 | 0.029 |
340
- | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.994 | 0.856 | 0.856 |
341
- | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 1.000 | 0.897 | 0.897 |
342
- | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.935 | 0.935 | 0.751 | 0.751 |
343
- | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.998 | 0.908 | 0.908 |
344
- | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.805 | 0.000 | 0.000 |
345
- | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.991 | 0.887 | 0.887 |
346
- | `kas_Deva` | Kashmiri | 47 | 0.557 | 0.703 | 0.703 | 0.587 | 0.587 |
347
- | `cjk_Latn` | Chokwe | 24 | 0.857 | 1.000 | 1.000 | 0.807 | 0.807 |
348
- | `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.961 | 0.959 | 0.894 | 0.894 |
349
- | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.981 | 0.854 | 0.854 |
350
- | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 1.000 | 0.772 | 0.772 |
351
- | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 1.000 | 0.956 | 0.956 |
352
- | `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.854 | 0.854 |
353
- | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 1.000 | 0.838 | 0.838 |
354
-
355
- Per-language precision, recall (sensitivity), F1 and specificity for every model are in `reports/<set>/`.
356
-
357
- ## Training
358
-
359
- 1,943,957 passages, best validation macro F1 at step 106000, 0.6 hours.
360
- Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character
361
- snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: pytorch
4
+ pipeline_tag: text-classification
5
+ tags:
6
+ - language-identification
7
+ - onnx
8
+ - byte-level
9
+ datasets:
10
+ - baobabtech/millid-dataset
11
+ metrics:
12
+ - f1
13
+ - accuracy
14
+ model-index:
15
+ - name: millid-tiny-v1.5
16
+ results:
17
+ - task:
18
+ type: text-classification
19
+ name: Language identification
20
+ dataset:
21
+ name: MillID test split (the full build)
22
+ type: baobabtech/millid-dataset
23
+ split: test
24
+ metrics:
25
+ - type: f1
26
+ name: Macro F1
27
+ value: 0.9554
28
+ - type: accuracy
29
+ name: Top-1 accuracy
30
+ value: 0.9587
31
+ - task:
32
+ type: text-classification
33
+ name: Language identification
34
+ dataset:
35
+ name: UDHR-LID, the languages MillID knows
36
+ type: baobabtech/millid-dataset
37
+ config: udhr
38
+ split: test
39
+ metrics:
40
+ - type: f1
41
+ name: Macro F1
42
+ value: 0.8764
43
+ - type: accuracy
44
+ name: Top-1 accuracy
45
+ value: 0.8492
46
+ ---
47
+
48
+ # MillID tiny v1.5: MultiLingual Language IDentifier
49
+
50
+ A byte-level, encoder-only transformer that names the language of a passage, from a table cell to a paragraph: **1,641 language-script labels** (ISO 639-3 + script, e.g. `eng_Latn`, `amh_Ethi`). 2.9M parameters, trained from scratch. Apache-2.0.
51
+
52
+ ## The MillID models
53
+
54
+ | Model | Size | Macro F1, `test` / `udhr` | Runs best on |
55
+ |---|---|---|---|
56
+ | **MillID tiny v1.5** (this model) | 2.9M parameters, ONNX 11 MB | 0.9554 / 0.8764 | a CPU, or a web page |
57
+ | [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M parameters, ONNX 455 MB | 0.9576 / 0.8780 | a GPU |
58
+ | [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M parameters, ONNX 764 MB | 0.9646 / 0.8812 | a GPU: the most accurate |
59
+
60
+ All three, language by language, against GlotLID and Meta's fastText: the [leaderboard](https://huggingface.co/baobabtech/millid). GlotLID, restricted to MillID's labels, scores 0.9755 / 0.8873 on the same sets.
61
+
62
+ ## Use it
63
+
64
+ No server and no PyTorch: the model runs in your process from `onnx/model.onnx`, in Python or JavaScript, and the two give the same answers (one encoder, ported line for line and tested against the same fixtures in [baobab-tech/millid](https://github.com/baobab-tech/millid)).
65
+
66
+ ```bash
67
+ pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git" # [gpu] for CUDA
68
+ ```
69
+
70
+ ```python
71
+ from millid import MillID
72
+
73
+ model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
74
+ model.predict("Lave men ou ak savon anvan ou manje.", k=3)
75
+ # [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
76
+ model.predict(["one text", "another"], k=3) # one list of answers per text
77
+ model.identify_document(long_text) # any length: its languages and segments
78
+ ```
79
+
80
+ ```bash
81
+ millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3
82
+ ```
83
+
84
+ JavaScript, in Node or a browser (`js/` in the same repository; not on npm yet):
85
+
86
+ ```js
87
+ import { MillID } from "@baobabtech/millid"; // "@baobabtech/millid/web" in a browser, with { ort }
88
+ const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
89
+ await model.predict("Lave men ou ak savon anvan ou manje.", 3);
90
+ ```
91
+
92
+ `millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.
93
+
94
+ The ONNX graph takes, for each UTF-8 byte of a passage cut to 255 bytes after a CLS token, `ids`, `script` and `charpos` (int64, `[batch, bytes]`), `ngrams` (int64, `[batch, bytes, 3]`) and `mask` (bool, `[batch, bytes]`), and returns `logits` (float32, `[batch, labels]`). `millid.encode_batch` and the JavaScript `encodeBatch` build them; script ids come from a frozen table (Unicode 15.0), so every runtime gives the same ones.
95
+
96
+ ## Architecture
97
+
98
+ UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 bytes per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. Up to 256 bytes per passage. See `ARCHITECTURE.md`.
99
+
100
+ ## Evaluation against GlotLID and fastText
101
+
102
+ `validation` chose the checkpoint; `test` is split by document from training; `udhr` is the Universal Declaration of Human Rights, labelled by its translators, over the languages MillID knows, and `udhr-all` over all of them. `glotlid` is [GlotLID](https://huggingface.co/cis-lmu/glotlid) as it is; `glotlid@millid` is GlotLID restricted to MillID's labels, the like-for-like comparison. GlotLID's own labels came from training on data that includes UDHR and labelled FinePDFs and FineWeb-2, so its scores on `udhr` and on those sources are optimistic. `fasttext` is Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), its answers mapped onto MillID's spelling (Simplified and Traditional Han as `Hani`, `pes`/`prs` as `fas`, `est` as `ekk`). It cannot answer most of the 1,600 languages here, so `test-common` and `udhr-common` score every model on the passages whose language all of them know: the head-to-head on equal terms.
103
+
104
+ | Set | Model | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 | Languages |
105
+ |---|---|---|---|---|---|---|---|---|---|
106
+ | validation | millid | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 | 1626 |
107
+ | test | millid | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 | 1621 |
108
+ | test | glotlid | 0.9776 | 0.9977 | 0.9984 | 0.9783 | 0.9752 | 0.9748 | 1.0000 | 1621 |
109
+ | test | glotlid@millid | 0.9783 | 0.9978 | 0.9985 | 0.9778 | 0.9765 | 0.9755 | 1.0000 | 1621 |
110
+ | test | fasttext | 0.2511 | 0.2610 | 0.2625 | 0.0520 | 0.1205 | 0.0657 | 0.0000 | 1621 |
111
+ | test | fasttext@millid | 0.2515 | 0.2611 | 0.2625 | 0.0521 | 0.1209 | 0.0659 | 0.0000 | 1621 |
112
+ | test-common | millid | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 | 213 |
113
+ | test-common | glotlid | 0.9738 | 0.9925 | 0.9946 | 0.9924 | 0.9771 | 0.9835 | 0.9983 | 213 |
114
+ | test-common | glotlid@millid | 0.9749 | 0.9927 | 0.9948 | 0.9923 | 0.9790 | 0.9846 | 0.9983 | 213 |
115
+ | test-common | fasttext | 0.9339 | 0.9707 | 0.9762 | 0.9357 | 0.9169 | 0.9149 | 0.9680 | 213 |
116
+ | test-common | fasttext@millid | 0.9352 | 0.9710 | 0.9763 | 0.9354 | 0.9201 | 0.9173 | 0.9680 | 213 |
117
+ | udhr | millid | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 | 358 |
118
+ | udhr | glotlid | 0.8598 | 0.9280 | 0.9421 | 0.9261 | 0.8826 | 0.8865 | 0.9915 | 358 |
119
+ | udhr | glotlid@millid | 0.8614 | 0.9294 | 0.9433 | 0.9258 | 0.8844 | 0.8873 | 0.9915 | 358 |
120
+ | udhr | fasttext | 0.4625 | 0.4857 | 0.4903 | 0.3152 | 0.4479 | 0.3497 | 0.0000 | 358 |
121
+ | udhr | fasttext@millid | 0.4629 | 0.4857 | 0.4904 | 0.3152 | 0.4483 | 0.3499 | 0.0000 | 358 |
122
+ | udhr-common | millid | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 | 170 |
123
+ | udhr-common | glotlid | 0.9336 | 0.9678 | 0.9732 | 0.9888 | 0.9576 | 0.9657 | 1.0000 | 170 |
124
+ | udhr-common | glotlid@millid | 0.9338 | 0.9683 | 0.9736 | 0.9888 | 0.9579 | 0.9659 | 1.0000 | 170 |
125
+ | udhr-common | fasttext | 0.9187 | 0.9647 | 0.9741 | 0.9474 | 0.9431 | 0.9323 | 1.0000 | 170 |
126
+ | udhr-common | fasttext@millid | 0.9195 | 0.9647 | 0.9742 | 0.9474 | 0.9441 | 0.9329 | 1.0000 | 170 |
127
+ | udhr-all | millid | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 | 429 |
128
+ | udhr-all | glotlid | 0.7502 | 0.8145 | 0.8282 | 0.7695 | 0.7529 | 0.7454 | 0.9796 | 429 |
129
+ | udhr-all | glotlid@millid | 0.7378 | 0.7960 | 0.8079 | 0.7425 | 0.7381 | 0.7245 | 0.9748 | 429 |
130
+ | udhr-all | fasttext | 0.3961 | 0.4160 | 0.4200 | 0.2479 | 0.3737 | 0.2795 | 0.0000 | 429 |
131
+ | udhr-all | fasttext@millid | 0.3964 | 0.4160 | 0.4200 | 0.2479 | 0.3741 | 0.2797 | 0.0000 | 429 |
132
+
133
+ Top-1 accuracy by passage length:
134
+
135
+ | Set | Model | short | sentence | paragraph |
136
+ |---|---|---|---|---|
137
+ | validation | millid | 0.9312 | 0.9788 | 0.9850 |
138
+ | test | millid | 0.9333 | 0.9797 | 0.9833 |
139
+ | test | glotlid | 0.9664 | 0.9853 | 0.9911 |
140
+ | test | glotlid@millid | 0.9674 | 0.9858 | 0.9912 |
141
+ | test | fasttext | 0.2760 | 0.2501 | 0.1958 |
142
+ | test | fasttext@millid | 0.2764 | 0.2506 | 0.1959 |
143
+ | test-common | millid | 0.9125 | 0.9685 | 0.9725 |
144
+ | test-common | glotlid | 0.9594 | 0.9876 | 0.9947 |
145
+ | test-common | glotlid@millid | 0.9609 | 0.9885 | 0.9949 |
146
+ | test-common | fasttext | 0.9112 | 0.9562 | 0.9651 |
147
+ | test-common | fasttext@millid | 0.9125 | 0.9581 | 0.9655 |
148
+
149
+ ### Leaderboard: test
150
+
151
+ MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText restricted to MillID's labels, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. The full table, with precision and recall, is `reports/test/leaderboard.csv`.
152
+
153
+ The 20 languages with the most passages:
154
+
155
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
156
+ |---|---|---|---|---|---|---|---|
157
+ | `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.928 | 0.808 | 0.808 |
158
+ | `eng_Latn` | English | 1444 | 0.917 | 0.858 | 0.859 | 0.653 | 0.648 |
159
+ | `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.951 | 0.574 | 0.574 |
160
+ | `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.993 | 0.965 | 0.965 |
161
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.785 | 0.513 | 0.512 |
162
+ | `por_Latn` | Portuguese | 657 | 0.947 | 0.947 | 0.949 | 0.756 | 0.755 |
163
+ | `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.993 | 0.922 | 0.922 |
164
+ | `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.896 | 0.539 | 0.539 |
165
+ | `swh_Latn` | Swahili | 592 | 0.848 | 0.917 | 0.916 | 0.346 | 0.346 |
166
+ | `mar_Deva` | Marathi | 589 | 0.965 | 0.989 | 0.990 | 0.926 | 0.926 |
167
+ | `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.956 | 0.425 | 0.425 |
168
+ | `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.999 | 0.232 | 0.232 |
169
+ | `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.985 | 0.861 | 0.860 |
170
+ | `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.994 | 0.951 | 0.951 |
171
+ | `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.949 | 0.462 | 0.462 |
172
+ | `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.948 | 0.949 | 0.000 | 0.000 |
173
+ | `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.991 | 0.994 | 0.976 | 0.976 |
174
+ | `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.977 | 0.978 | 0.660 | 0.660 |
175
+ | `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.969 | 0.915 | 0.914 |
176
+ | `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.986 | 0.000 | 0.000 |
177
+
178
+ Where MillID gains most over GlotLID:
179
+
180
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
181
+ |---|---|---|---|---|---|---|---|
182
+ | `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 | 0.000 | 0.000 |
183
+ | `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.000 |
184
+ | `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.581 | 0.000 | 0.000 |
185
+ | `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.621 | 0.000 | 0.000 |
186
+ | `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.651 | 0.000 | 0.000 |
187
+ | `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.667 | 0.000 | 0.000 |
188
+ | `und_Geor` | und | 32 | 0.877 | 0.694 | 0.694 | 0.000 | 0.000 |
189
+ | `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.761 | 0.000 | 0.000 |
190
+ | `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.822 | 0.000 | 0.000 |
191
+ | `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.786 | 0.000 | 0.000 |
192
+ | `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.800 | 0.000 | 0.000 |
193
+ | `und_Kana` | und | 136 | 0.982 | 0.862 | 0.862 | 0.000 | 0.000 |
194
+ | `und_Yiii` | und | 38 | 1.000 | 0.894 | 0.884 | 0.000 | 0.000 |
195
+ | `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.625 | 0.842 | 0.000 | 0.000 |
196
+ | `dag_Latn` | Dagbani | 149 | 0.767 | 0.662 | 0.660 | 0.000 | 0.000 |
197
+
198
+ Where MillID loses most to GlotLID:
199
+
200
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
201
+ |---|---|---|---|---|---|---|---|
202
+ | `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 1.000 | 0.000 | 0.000 |
203
+ | `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.946 | 0.000 | 0.000 |
204
+ | `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 1.000 | 0.000 | 0.000 |
205
+ | `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.823 | 0.825 | 0.007 | 0.007 |
206
+ | `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
207
+ | `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 1.000 | 0.000 | 0.000 |
208
+ | `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.783 | 0.000 | 0.000 |
209
+ | `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.944 | 0.178 | 0.178 |
210
+ | `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 1.000 | 0.000 | 0.000 |
211
+ | `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.902 | 0.181 | 0.181 |
212
+ | `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 1.000 | 0.000 | 0.000 |
213
+ | `nyf_Latn` | Giryama | 10 | 0.645 | 0.952 | 0.870 | 0.000 | 0.000 |
214
+ | `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.992 | 0.000 | 0.000 |
215
+ | `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.824 | 0.000 | 0.000 |
216
+ | `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 1.000 | 0.000 | 0.000 |
217
+
218
+ ### Leaderboard: udhr
219
+
220
+ MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText restricted to MillID's labels, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. The full table, with precision and recall, is `reports/udhr/leaderboard.csv`.
221
+
222
+ The 20 languages with the most passages:
223
+
224
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
225
+ |---|---|---|---|---|---|---|---|
226
+ | `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.999 | 0.946 | 0.946 |
227
+ | `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.376 | 0.435 | 0.435 |
228
+ | `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.868 | 0.468 | 0.468 |
229
+ | `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.011 | 0.000 | 0.000 |
230
+ | `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 | 0.000 | 0.000 |
231
+ | `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.854 | 0.000 | 0.000 |
232
+ | `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.986 | 0.000 | 0.000 |
233
+ | `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 1.000 | 0.876 | 0.876 |
234
+ | `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
235
+ | `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.877 | 0.000 | 0.000 |
236
+ | `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 1.000 | 0.944 | 0.944 |
237
+ | `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.514 | 0.000 | 0.000 |
238
+ | `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.680 | 0.000 | 0.000 |
239
+ | `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.992 | 0.000 | 0.000 |
240
+ | `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.979 | 0.699 | 0.699 |
241
+ | `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 1.000 | 0.806 | 0.806 |
242
+ | `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 1.000 | 0.877 | 0.877 |
243
+ | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
244
+ | `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.996 | 0.399 | 0.396 |
245
+ | `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.663 | 0.000 | 0.000 |
246
+
247
+ Where MillID gains most over GlotLID:
248
+
249
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
250
+ |---|---|---|---|---|---|---|---|
251
+ | `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.427 | 0.000 | 0.000 |
252
+ | `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.167 | 0.000 | 0.000 |
253
+ | `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.483 | 0.000 | 0.000 |
254
+ | `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.578 | 0.189 | 0.189 |
255
+ | `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.500 | 0.000 | 0.000 |
256
+ | `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.154 | 0.000 | 0.000 |
257
+ | `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.462 | 0.000 | 0.000 |
258
+ | `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.731 | 0.000 | 0.000 |
259
+ | `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.812 | 0.966 | 0.966 |
260
+ | `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.580 | 0.111 | 0.111 |
261
+ | `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 | 0.000 | 0.000 |
262
+ | `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.739 | 0.000 | 0.000 |
263
+ | `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 | 0.000 | 0.000 |
264
+ | `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.859 | 0.487 | 0.487 |
265
+ | `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.098 | 0.000 | 0.000 |
266
+
267
+ Where MillID loses most to GlotLID:
268
+
269
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
270
+ |---|---|---|---|---|---|---|---|
271
+ | `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.990 | 0.000 | 0.000 |
272
+ | `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.912 | 0.000 | 0.000 |
273
+ | `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.434 | 0.015 | 0.015 |
274
+ | `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.378 | 0.000 | 0.000 |
275
+ | `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.882 | 0.874 | 0.000 | 0.000 |
276
+ | `idu_Latn` | Idoma | 51 | 0.603 | 0.879 | 0.938 | 0.000 | 0.000 |
277
+ | `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 | 1.000 | 1.000 |
278
+ | `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.906 | 0.000 | 0.000 |
279
+ | `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.974 | 0.000 | 0.000 |
280
+ | `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.460 | 0.000 | 0.000 |
281
+ | `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.957 | 0.975 | 0.975 |
282
+ | `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.727 | 0.474 | 0.474 |
283
+ | `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 1.000 | 0.389 | 0.389 |
284
+ | `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.963 | 0.000 | 0.000 |
285
+ | `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.857 | 0.000 | 0.000 |
286
+
287
+ ### Leaderboard: test-common
288
+
289
+ MillID's F1 against the others', language by language: against GlotLID restricted to MillID's labels, MillID is better on **6** languages, worse on **154** and within 0.01 on 53; against fastText restricted to MillID's labels, MillID is better on **85** languages, worse on **51** and within 0.01 on 77. The full table, with precision and recall, is `reports/test-common/leaderboard.csv`.
290
+
291
+ The 20 languages with the most passages:
292
+
293
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
294
+ |---|---|---|---|---|---|---|---|
295
+ | `fra_Latn` | French | 1652 | 0.943 | 0.955 | 0.955 | 0.963 | 0.963 |
296
+ | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
297
+ | `spa_Latn` | Spanish | 1136 | 0.937 | 0.958 | 0.958 | 0.941 | 0.941 |
298
+ | `tam_Taml` | Tamil | 754 | 0.997 | 1.000 | 1.000 | 0.997 | 0.997 |
299
+ | `arb_Arab` | Arabic (Modern Standard) | 745 | 0.871 | 0.886 | 0.886 | 0.957 | 0.954 |
300
+ | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
301
+ | `kan_Knda` | Kannada | 601 | 0.973 | 0.994 | 0.994 | 0.999 | 0.999 |
302
+ | `zul_Latn` | Zulu | 594 | 0.883 | 0.904 | 0.904 | 0.970 | 0.970 |
303
+ | `swh_Latn` | Swahili | 592 | 0.893 | 0.954 | 0.953 | 0.945 | 0.945 |
304
+ | `mar_Deva` | Marathi | 589 | 0.968 | 0.989 | 0.990 | 0.987 | 0.987 |
305
+ | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
306
+ | `hun_Latn` | Hungarian | 576 | 0.989 | 1.000 | 1.000 | 0.980 | 0.980 |
307
+ | `afr_Latn` | Afrikaans | 569 | 0.972 | 0.985 | 0.985 | 0.986 | 0.986 |
308
+ | `tel_Telu` | Telugu | 555 | 0.997 | 0.998 | 0.998 | 0.999 | 0.999 |
309
+ | `rus_Cyrl` | Russian | 528 | 0.945 | 0.985 | 0.985 | 0.927 | 0.927 |
310
+ | `ory_Orya` | Odia (individual language) | 511 | 0.994 | 0.997 | 1.000 | 0.996 | 0.996 |
311
+ | `sot_Latn` | Southern Sotho | 491 | 0.969 | 0.977 | 0.978 | 0.856 | 0.856 |
312
+ | `nso_Latn` | Northern Sotho | 479 | 0.958 | 0.970 | 0.970 | 0.967 | 0.967 |
313
+ | `asm_Beng` | Assamese | 467 | 0.997 | 0.999 | 0.999 | 0.997 | 0.997 |
314
+ | `ben_Beng` | Bangla | 466 | 0.988 | 0.997 | 0.997 | 0.989 | 0.989 |
315
+
316
+ Where MillID gains most over GlotLID:
317
+
318
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
319
+ |---|---|---|---|---|---|---|---|
320
+ | `ewo_Latn` | Ewondo | 134 | 0.959 | 0.879 | 0.883 | 0.860 | 0.860 |
321
+ | `eng_Latn` | English | 1444 | 0.932 | 0.876 | 0.877 | 0.857 | 0.857 |
322
+ | `arb_Latn` | Standard Arabic | 163 | 0.942 | 0.906 | 0.913 | 0.852 | 0.852 |
323
+ | `tso_Latn` | Tsonga | 581 | 0.979 | 0.956 | 0.956 | 0.986 | 0.986 |
324
+ | `fuv_Latn` | Nigerian Fulfulde | 164 | 0.907 | 0.877 | 0.892 | 0.859 | 0.859 |
325
+ | `knc_Latn` | Central Kanuri | 293 | 0.967 | 0.952 | 0.952 | 0.981 | 0.981 |
326
+ | `quy_Latn` | Ayacucho Quechua | 271 | 0.940 | 0.925 | 0.933 | 0.979 | 0.979 |
327
+ | `kam_Latn` | Kamba | 272 | 0.980 | 0.972 | 0.974 | 0.863 | 0.863 |
328
+ | `bjn_Arab` | Banjar | 116 | 0.992 | 0.834 | 0.987 | 0.453 | 0.960 |
329
+ | `por_Latn` | Portuguese | 657 | 0.959 | 0.954 | 0.956 | 0.950 | 0.949 |
330
+ | `mlt_Latn` | Maltese | 291 | 0.995 | 0.993 | 0.993 | 0.961 | 0.961 |
331
+ | `hye_Armn` | Armenian | 178 | 1.000 | 1.000 | 1.000 | 0.977 | 0.977 |
332
+ | `khm_Khmr` | Khmer | 187 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
333
+ | `guj_Gujr` | Gujarati | 354 | 1.000 | 1.000 | 1.000 | 0.984 | 0.984 |
334
+ | `mal_Mlym` | Malayalam | 338 | 1.000 | 1.000 | 1.000 | 0.994 | 0.994 |
335
+
336
+ Where MillID loses most to GlotLID:
337
+
338
+ | Label | Language | Passages | MillID F1 | GlotLID F1 | GlotLID F1, restricted | fastText F1 | fastText F1, restricted |
339
+ |---|---|---|---|---|---|---|---|
340
+ | `knc_Arab` | Central Kanuri | 68 | 0.522 | 0.857 | 0.860 | 0.029 | 0.029 |
341
+ | `hrv_Latn` | Croatian | 168 | 0.659 | 0.994 | 0.994 | 0.856 | 0.856 |
342
+ | `bos_Latn` | Bosnian | 198 | 0.685 | 1.000 | 1.000 | 0.897 | 0.897 |
343
+ | `tzm_Tfng` | Central Atlas Tamazight | 157 | 0.697 | 0.934 | 0.934 | 0.751 | 0.751 |
344
+ | `zsm_Latn` | Standard Malay | 244 | 0.789 | 0.998 | 0.998 | 0.908 | 0.908 |
345
+ | `taq_Tfng` | Tamasheq | 104 | 0.654 | 0.805 | 0.805 | 0.000 | 0.000 |
346
+ | `ind_Latn` | Indonesian | 335 | 0.842 | 0.991 | 0.991 | 0.887 | 0.887 |
347
+ | `kas_Deva` | Kashmiri | 47 | 0.557 | 0.703 | 0.703 | 0.587 | 0.587 |
348
+ | `cjk_Latn` | Chokwe | 24 | 0.857 | 1.000 | 1.000 | 0.807 | 0.807 |
349
+ | `nob_Latn` | Norwegian Bokmål | 265 | 0.821 | 0.961 | 0.959 | 0.894 | 0.894 |
350
+ | `kin_Latn` | Kinyarwanda | 210 | 0.849 | 0.981 | 0.981 | 0.854 | 0.854 |
351
+ | `mag_Deva` | Magahi | 97 | 0.890 | 1.000 | 1.000 | 0.772 | 0.772 |
352
+ | `lim_Latn` | Limburgish | 196 | 0.896 | 1.000 | 1.000 | 0.956 | 0.956 |
353
+ | `run_Latn` | Rundi | 291 | 0.883 | 0.985 | 0.985 | 0.855 | 0.855 |
354
+ | `taq_Latn` | Tamasheq | 106 | 0.902 | 1.000 | 1.000 | 0.838 | 0.838 |
355
+
356
+ Per-language precision, recall (sensitivity), F1 and specificity for every model are in `reports/<set>/`.
357
+
358
+ ## Training
359
+
360
+ 1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.
361
+
362
+ The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages.
363
+
364
+ ## Licence
365
+
366
+ The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).