mutkuoz/turkic-relevance-laya-multilingual

Turkic relevance classifier (Laya-multilingual, fine-tuned)

Given a Turkish Wikipedia article (its title, categories and opening paragraphs), this model says how it relates to the Turkic world: core, modern or unrelated. It was built for Project: Turkic Blue, a community encyclopedia of Turkic history and culture, to find the Turk-related part of Turkish Wikipedia: it judged ~59,000 articles in a crawl and selected 14,244 as core.

It's Laya-multilingual's encoder with a new three-label head, so it answers this one question. It doesn't support Laya's general question-answer API.

Labels

Label	Meaning	Examples
core	Turkic heritage: history of Turkic peoples and states (Ottomans and Seljuks included), peoples and tribes, languages and scripts, mythology, religion, literature, arts, traditions, historical and cultural figures	Kutadgu Bilig, II. Mehmed, Göktürkler, Nevruz, an Ottoman mosque, Âşık Veysel
modern	Present-day Turkey or Turkic countries without heritage value: current politics, sport, film and TV, settlements, companies, transport	a political party, a football club, a district, a TV series
unrelated	Not about Turks or the Turkic world	Moscow, asphalt, the turkey (bird)

Where core ends and modern begins is an editorial choice, made for turkic.blue.

Usage

pip install torch "transformers>=5" huggingface_hub safetensors

from huggingface_hub import hf_hub_download
import importlib.util

repo = "mutkuoz/turkic-relevance-laya-multilingual"
spec = importlib.util.spec_from_file_location("turkic_relevance", hf_hub_download(repo, "turkic_relevance.py"))
turkic_relevance = importlib.util.module_from_spec(spec)
spec.loader.exec_module(turkic_relevance)

model = turkic_relevance.TurkicRelevance(repo)  # GPU if available, else CPU
model.predict({
    "title": "Kutadgu Bilig",
    "categories": ["Karahanlılar", "Türk edebiyatı eserleri"],
    "text": "Kutadgu Bilig, Yusuf Has Hacib'in 1069-1070 yıllarında Karahanlılar döneminde yazdığı ...",
})
# {'core': 0.98..., 'modern': 0.0..., 'unrelated': 0.0...}

turkic_relevance.py is about 60 lines of plain transformers: the encoder is a standard ModernBertModel (AutoModel.from_pretrained works), mean-pooled over tokens, then the linear head in head.safetensors. The input is formatted as title, a line of up to 15 categories joined by ; , a blank line, then the opening text, truncated to 384 tokens. Categories are optional; the model was trained to work without them too.

Use p(core) >= 0.5 as a starting threshold; raise it (0.8–0.9) where precision matters more than recall.

How it was made

Base. Laya-multilingual (Convai Innovations, Apache-2.0), built on mmBERT-base (JHU CLSP, MIT). We kept Laya's encoder (ModernBERT, 22 layers, 307M parameters) and replaced its decision head with mean pooling and a linear layer (2,307 parameters).

Why fine-tune. Zero-shot, Laya-multilingual was at chance on this task, for every phrasing we tried (three-way and yes/no questions, in English and Turkish): 20–45% accuracy, and its probability of "about the Turks" was no better than the base rate at any threshold. Laya's own card says the base checkpoints are a base to specialise, not a zero-shot decision engine.

Training data. 27,000 Turkish Wikipedia articles (dump of late September 2026), 9,000 per label, drawn mostly from the link neighbourhood of 150 Turkic seed articles plus random articles. Labels came from the articles' Wikipedia categories via hand-written rules, which agree with the hand labels below 92.6% of the time on the 78% of articles they cover. Two refinements came from error analysis:

• "Unrelated" is judged from categories, not from whether the text mentions Turks, so articles that only mention Turkish in passing (etymology, translations) are negatives.
• Republic-era state and public life (laws, ministries, railways, recent military operations) is labelled modern, which stopped the classifier drifting into it.

The article text is CC BY-SA 4.0 and is not redistributed here.

Training. AdamW, learning rate 2e-5 for the encoder and 4e-4 for the head, one-cycle schedule, batch 16, label smoothing 0.05 (the labels are noisy), categories hidden for 40% of examples, 384 tokens, bf16. Two epochs on a laptop RTX 5070 Ti, about 8 minutes each; the first epoch scored best on the hand labels and is the one released.

Evaluation

Two hand-labelled sets, never used for training. The labels were made with an AI assistant (Claude) from each article's title, categories and opening text, then reviewed and accepted by the turkic.blue content lead. Judgement calls are marked "borderline" in the files (eval/).

Test set	Accuracy (3-way)	core precision / recall at p ≥ 0.5	at p ≥ 0.9
296 articles one link from the seeds (eval/labels.tsv)	0.858	0.859 / 0.915	0.891 / 0.856
69 articles two to three links out (eval/drift_labels.tsv)*	0.884	0.889 / 0.930	0.902 / 0.860
Same 296, base Laya-multilingual zero-shot	0.20–0.45	≈ base rate

* Drawn from articles that an earlier version of this model called core, to test whether this version fixed its drift. Not a random sample.

In use: a crawl of Turkish Wikipedia judged 59,260 articles. In a random sample of 50 articles it selected as core, 39 were clearly right, 4 borderline and 7 wrong.

Limitations

• Trained and tested only on Turkish Wikipedia articles. Other kinds of text, and other languages, are untested.
• The labels inherit the reach and biases of Wikipedia's categories: Turkey-centric and heavy on Ottoman topics. Turkic peoples with thinner coverage on Turkish Wikipedia (e.g. Siberian Turkic peoples) are less represented.
• Known errors: some Republic-era institutions and transport (e.g. city rail stations) still come out core; people and places next to Turkic history (Saladin, Byzantine and Balkan royals) are sometimes core; 20th-century Turkish writers sit on the line between core and modern.
• core means "about the Turkic world", not a judgement on the article's content. On contested history, the model says nothing about which account is right.

License and credits

Apache-2.0, like Laya. Built on Laya-multilingual by Convai Innovations (Apache-2.0) and mmBERT-base by JHU CLSP (MIT); please credit both. Trained on text from Turkish Wikipedia (CC BY-SA 4.0); the evaluation files list Wikipedia article titles and page ids.