Multi-corpus hate speech ensemble

Binary classifier: 1 = abusive or hateful, 0 = neither. Trained on ten public corpora in English, Spanish, French, Arabic and Roman Urdu (Davidson, HateXplain, DynaHate, ETHOS, HatEval EN/ES, RUHSOLD, MLMA EN/FR/AR). The ensemble weights and threshold were chosen with HABC on validation data (minimax objective).

Test results (5,874 texts, never used for any choice)

accuracy 0.7673 | macro-F1 0.7386 | worst-corpus macro-F1 0.5312 | ROC-AUC 0.8264

corpus n accuracy macro-F1
davidson2017 896 0.9118 0.8486
dynabench2021 900 0.6044 0.6042
ethos2022 150 0.6667 0.6563
hateval2019_en 150 0.6867 0.6791
hateval2019_es 75 0.6267 0.5879
hatexplain2021 899 0.6841 0.6714
mlma2019_ar 492 0.7764 0.7180
mlma2019_en 819 0.8449 0.5312
mlma2019_fr 593 0.7707 0.6035
romanurdu2020 900 0.8333 0.8333

Use

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("Akash212/multicorpus-hate-speech-ensemble", token="<your token>")
sys.path.insert(0, path)
from predict import HateEnsemble
model = HateEnsemble(path)
model.predict(["text one", "text two"])

Limitations

Research model. Labels from different corpora follow different definitions, and hate and offensive language are merged. Scores drop on data unlike the training corpora. Use it to help human moderators, not to remove content automatically. Some training corpora allow research use only.

Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support