preBIJ :: Bosnian / Croatian / Serbian LID

The linguistic intelligence of a native BCS language model used for language identificaton.

predBor Identifikator Jezika ("predBor Language Identifier"), a language classification model based on the now retired predBor-v1 small BCS language model, made specifically to tell apart Bosnian, Croatian and Serbian.

Retired but not forgotten

As noted, I made the decision to retire the predBor model series as the newer Bor models are in the works. However, retirement does not necessarily mean abandonment, at least not to me.

preBIJ is a direct fine-tune of predBor, changing the architecture to a classification model while preserving all linguistical knowledge of the base, effectively giving the old model a new life.

Architecture

Name Value
Architecture LlamaForSequenceClassification
Base Model absltnull/predBor-v1
Maximum Sequence Length 4096 tokens (trained up to 196 tokens, retains accuracy up to 1024, untested beyond)
Classes bs, hr, sr
Tokenizer Vocabulary 65,000 tokens
Tokenizer BorTokenizer-v2
Supported Scripts Bosnian (Lat, Cyr), Serbian (Lat, Cyr), Croatian (Lat)

The Idea

After the retirement of predBor, new ideas began to emerge for its new purpose. RoBERTo-v2, while capable, is severely limited by its tokenizer and size, so the natural question would be: how would a native model perform at labeling such languages?

predBor was the perfect match. It had a native BCS tokenizer, was triple the size of RoBERTo-v2, and had a massive, specialized understanding of all three languages (plus English, not relevant), so it was naturally the first choice.

The Surgery

I started with the original predBor-v1 model. Instead of loading it as a casual LM model, I used the AutoModelForSequenceClassification.from_pretrained("absltnull/predBor-v1", num_labels=3) function to load it as a three-way classification model with the type LlamaForSequenceClassification, initializing a random classification head on top of the model, instead of the base LM head.

Next, I had to train that randomly-initialized head to actually know which language is which, using the representations present in the base LM. This was a trivial task for me, though, as I already have my absltnull/bcs-lang-id dataset with over 90k examples of labeled BCS data.

The model trained for about 4 hours on my RTX 2050, and training loss dropped under 1.0 in the first 4% of the run, indicating that the head "got the gist" in like 10 minutes.

The Performance

The final training loss was 0.2, with 92% accuracy on the evaluation set, basically beating RoBERTo-v2 by 5 points of accuracy and F1.

I ranked RoBERTo-v2 vs preBIJ on multiple tests other than accuracy and F1, and here are the results:

Round 1: Accuracy

Plain

model (acc, f1): preBIJ (0.919, 0.919) vs RoBERTo-v2 (0.868, 0.868)

A close call, but actually beats RoBERTo-v2 on its own dataset. preBIJ was actually trained on the same dataset as RoBERTo-v2, yet it came out as the victor in this test.

Round 2: Cyrillic

Cyrillic

Bosnian script (prebij acc, roberto acc): Latin (0.892, 0.847), Cyrillic (0.792, 0.000)
Serbian script (prebij acc, roberto acc): Latin (0.950, 0.871), Cyrillic (0.959, 1.000)

As we all know, RoBERTo-v2 has a massive Cyrillic bias: "all Cyrillic text => 100% Serbian". This causes it to label every single Cyrillic text as Serbian, which makes it score 0% on the Cyrillic Bosnian test, and score a 100% on the Cyrillic Serbian test.

preBIJ, on the other hand, has managed to outperform RoBERTo-v2 on every metric on this test, scoring:

  • 90% accuracy in recognizing Bosnian Latin text,
  • 80% accuracy in recognizing Bosnian Cyrillic text,
  • 95% accuracy in recognizing Serbian Latin text,
  • 96% accuracy in recognizing Serbian Cyrillic text

The amazing part is: it was never specifically trained to do this.

preBIJ has never seen a single sample in Cyrillic, but predBor has. This means that the classification head has learned to generalize, basically using predBor's diverse knowledge of all three languages to recognize the content of the input rather than finding shortcuts in data.

In plain words: predBor knows the difference between Bosnian and Serbian, no matter the script, and the classification head inherited that knowledge, making it beat RoBERTo-v2 in this metric effortlesly.

Round 3: Diacritics

Diacritics

script (prebij acc, roberto acc): Latin (0.921, 0.859), ASCII (0.921, 0.851)

Pretty useless test, but it proves a point: both models score the exact same result in labeling the languages, both with and without the diacritics (though RoBERTo-v2 lost like 0.008 points without diacritics, but we'll ignore that).

Using the model

After seeing these results, we are safe to conclude that preBIJ is the top model for distinguishing BCS languages, even in rare script cases.

To use the model, you can use this snipplet:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tok = AutoTokenizer.from_pretrained("absltnull/prebij")
model = AutoModelForSequenceClassification.from_pretrained("absltnull/prebij")

classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tok,
    device=0 if torch.cuda.is_available() else -1,
)

texts = [
    "Ove sedmice moram da kupim bijeli hljeb.", # bs
    "Ove nedelje moram da kupim beli hleb.", # sr
    "Ovog tjedna moram da kupim bijeli kruh.", # hr
]

predictions = classifier(texts)
for text, pred in zip(texts, predictions):
    print(f"[{pred['label']}] ({pred['score']:.2%})  {text}")

TL;DR

Absolute Intelligence has provided this model to be used freely, without any copyright obligations, other than stating that this model belongs to its rightful owner. Feel free to use, fine-tune or adapt this model to any of your preferences. Made with love in Bosnia. Bye.

Downloads last month
20
Safetensors
Model size
0.8B params
Tensor type
F32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for absltnull/prebij

Finetuned
(1)
this model

Dataset used to train absltnull/prebij