Multi-corpus hate speech ensemble
Binary classifier: 1 = abusive or hateful, 0 = neither. Trained on ten public corpora in English, Spanish, French, Arabic and Roman Urdu (Davidson, HateXplain, DynaHate, ETHOS, HatEval EN/ES, RUHSOLD, MLMA EN/FR/AR). The ensemble weights and threshold were chosen with HABC on validation data (minimax objective).
Test results (5,874 texts, never used for any choice)
accuracy 0.7673 | macro-F1 0.7386 | worst-corpus macro-F1 0.5312 | ROC-AUC 0.8264
| corpus | n | accuracy | macro-F1 |
|---|---|---|---|
| davidson2017 | 896 | 0.9118 | 0.8486 |
| dynabench2021 | 900 | 0.6044 | 0.6042 |
| ethos2022 | 150 | 0.6667 | 0.6563 |
| hateval2019_en | 150 | 0.6867 | 0.6791 |
| hateval2019_es | 75 | 0.6267 | 0.5879 |
| hatexplain2021 | 899 | 0.6841 | 0.6714 |
| mlma2019_ar | 492 | 0.7764 | 0.7180 |
| mlma2019_en | 819 | 0.8449 | 0.5312 |
| mlma2019_fr | 593 | 0.7707 | 0.6035 |
| romanurdu2020 | 900 | 0.8333 | 0.8333 |
Use
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("Akash212/multicorpus-hate-speech-ensemble", token="<your token>")
sys.path.insert(0, path)
from predict import HateEnsemble
model = HateEnsemble(path)
model.predict(["text one", "text two"])
Limitations
Research model. Labels from different corpora follow different definitions, and hate and offensive language are merged. Scores drop on data unlike the training corpora. Use it to help human moderators, not to remove content automatically. Some training corpora allow research use only.
- Downloads last month
- 2