tekkilab/pulaar-asr-v4

⚠️ INTERNAL — PRIVATE — SCHEDULED FOR DELETION

pulaar_asr_v4, uploaded for internal evaluation only. Do not make this repository public and do not redistribute the weights. Delete with teardown.sh when the evaluation is complete.

w2v-BERT 2.0 (facebook/w2v-bert-2.0, MIT), 606,729,025 parameters, fully fine-tuned with CTC over 1,025 SentencePiece classes, plus an n-gram language model rebuilt at load time from lm_train.tsv.gz — that file is not in this repository, see Caveats.

Measured results

WER %, frozen audio-de-duplicated test sets, per-domain language model:

Slice	WER	CER
Read Senegalese Pulaar	13.10	4.29
8 varieties	15.67	8.21
Spontaneous (WAXAL)	35.50	—
Radio (Kallaama)	41.61	—

Per variety: borgu 5.84 · caka 7.22 · bororro 10.36 · liptako 11.44 · maacina 12.09 · pulaar 13.10 · pular 19.97 · adamawa 40.51.

Use

import torch, soundfile as sf, sentencepiece as spm
from transformers import SeamlessM4TFeatureExtractor, Wav2Vec2BertForCTC

fe = SeamlessM4TFeatureExtractor.from_pretrained("facebook/w2v-bert-2.0")
model = Wav2Vec2BertForCTC.from_pretrained("<repo>/hf").eval()
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")

y, sr = sf.read("clip.wav", dtype="float32")          # 16 kHz mono
x = fe(y, sampling_rate=16000, return_tensors="pt")["input_features"]
ids = model(input_features=x).logits.argmax(-1)[0].tolist()
out, prev = [], -1
for i in ids:
    if i != prev and i != sp.vocab_size(): out.append(i)
    prev = i
print(sp.decode(out))

Greedy decoding as above loses 1–8 WER points depending on domain. For the language model use pulaar_asr.lm.beam with the weights in bundle.json (read 0.3/1.5, radio 0.6/3.0, spontaneous 0.3/0.5).

Caveats

• fp32, exactly as trained. No quantisation has been evaluated.

• Domain dominates variety: 13.1 on read speech, 41.6 on radio. The model has never seen noisy or reverberant training audio.

• Adamawa is weak (40.51) on 3.9 h of data and a different orthographic tradition.

• The read Senegalese test slice is 127 utterances, ≈ ±1.2 WER points.

• The companion private dataset is a 39 h subset that overlaps this model's training data — its test split is a diagnostic slice, not a benchmark.

• The weights derive from MIT-licensed w2v-BERT 2.0 but were trained on a corpus including share-alike and no-licence sources, and lm_train.tsv.gz contains corpus transcripts. That is why this repository is private.

• lm_train.tsv.gz is withheld. The language-model text is scraped from pulaar.org and is copyrighted; we do not redistribute it. Greedy decoding reproduces exactly from what is here; the LM-decoded figures above do not. Ask if you need the LM for a like-for-like comparison.

• The companion pulaar-eval-data test split is not a clean benchmark for this model. v4 was trained before the corpus was rebuilt, under different split boundaries, so some of the new test utterances were in v4's training data. It is a clean benchmark for any model trained on the new train split. The measured overlap is reported in that dataset's stats.json.