Instructions to use xincan/pubmedbert-semrep-accuracy-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use xincan/pubmedbert-semrep-accuracy-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier") model = AutoModelForSequenceClassification.from_pretrained("xincan/pubmedbert-semrep-accuracy-classifier", device_map="auto") - Notebooks
- Google Colab
- Kaggle
PubMedBERT SemRep accuracy classifier
This model is a reproduction of the PubMedBERT accuracy classifier described in Zhang R, Hristovski D, Schutte D, Kastrin A, Fiszman M, Kilicoglu H. Drug repurposing for COVID-19 via knowledge graph completion. Journal of Biomedical Informatics. 2021;115:103696. doi:10.1016/j.jbi.2021.103696; preprint arXiv:2010.09600 (methods in Section 3.2.1; results in Section 4.1 and Table 1). The model was trained on the annotated predications used in the paper, with the same pretrained model and the authors' training code. Averaged over 10 independently trained models, it achieved a test F1 score of 0.859, compared with 0.854 reported in the paper.
Model description
The model determines whether a semantic predication (subject, predicate, object) extracted by SemRep is correct, that is, whether it is stated in its source sentence. It is intended for removing incorrect predications from SemMedDB before a knowledge graph is constructed.
- Architecture: BERT-base with a sequence classification head (
BertForSequenceClassification) - Pretrained model: BiomedBERT (formerly PubMedBERT), uncased, pretrained on PubMed abstracts and full-text articles
- Labels:
CORRECT(1) andINCORRECT(0) - Language: English
- License: MIT, consistent with the pretrained model
Usage
Each input is a single text consisting of the source sentence, the [SEP] token, and the predication written as
subject predicate object. As in training, the text should be converted to lowercase, and every character other than
a letter or digit (including the underscore) should be replaced with a space.
import re
from transformers import pipeline
clf = pipeline("text-classification", model="xincan/pubmedbert-semrep-accuracy-classifier")
def norm(s):
return " ".join(re.sub(r"[\W_]+", " ", s).split()).lower()
def predication_input(sentence, subject, predicate, obj):
return f"{norm(sentence)} [SEP] {norm(f'{subject} {predicate} {obj}')}"
print(clf(predication_input(
"Pulmonary hypertension accompanying ventricular septal defect and patent ductus arteriosus.",
"Pulmonary hypertension", "COEXISTS_WITH", "ventricular septal defect")))
# [{'label': 'CORRECT', 'score': 0.978}]
print(clf(predication_input(
"Control of brain tryptophan concentration in rats on a high fat diet.",
"tryptophan", "TREATS", "rats")))
# [{'label': 'INCORRECT', 'score': 0.914}] (the sentence does not state that tryptophan treats rats)
Multiple predications can be classified at once by passing a list, for example
clf(texts, batch_size=64, truncation=True).
Training data
The training data comprise 7,038 SemRep predications from PubMed abstracts, each annotated as correct or incorrect with respect to its source sentence; 67.0% are correct. They cover 20 predicate types, the most frequent being INTERACTS_WITH (1,898), PROCESS_OF (775), STIMULATES (654), TREATS (649), INHIBITS (638), USES (454), COEXISTS_WITH (449), ADMINISTERED_TO (213), DIAGNOSES (191), and PRODUCES (173). The data combine the 6,492 annotations described in the paper (6,000 annotations from Vasilakes et al., JAMIA Open, 2018, and 492 newly annotated predications) and 546 additional annotations. The publicly available portion (6,492 predications) is provided in the kilicogluh/lbd-covid repository.
Training procedure
Training followed the authors' notebook
(COVID_CLF.ipynb).
- Data split: A test set of 10% was sampled with the original label distribution. The remaining data were balanced 1:1 by randomly removing correct predications and then divided into training (90%) and validation (10%) sets, yielding 3,762 training, 416 validation, and 703 test predications.
- Optimization: AdamW with a learning rate of 1e-5, weight decay of 0.01, and epsilon of 1e-6; a linear learning rate schedule with 150 warm-up steps; a batch size of 16; and cross-entropy loss. Training ran for at most 10 epochs, retaining the weights with the highest validation F1 score and stopping early after 3 epochs without improvement. The maximum sequence length was 512 tokens; the longest input contained 190 tokens.
- Model selection: Ten models were trained with the random seeds used in the authors' notebook. The released model is the one with the highest validation F1 score (seed 1386, epoch 4, validation F1 = 0.823).
- Hardware: A single NVIDIA A40 GPU.
Evaluation
Ten models were trained with the paper's method, each with a different random seed and a different random split of the data. Each model was evaluated on its own held-out test set of 703 predications (67% correct).
| Test set | Precision | Recall | F1 |
|---|---|---|---|
| Zhang et al. (2021), Table 1 | 0.816 | 0.895 | 0.854 |
| Mean of the 10 models (range) | 0.850 (0.798โ0.871) | 0.869 (0.815โ0.904) | 0.859 (0.839โ0.875) |
| Released model | 0.871 | 0.815 | 0.842 |
The mean performance of the 10 models is consistent with that reported in the paper. The released model is one of the 10 models; following the authors' code, it was selected by its F1 score on the validation set (0.823), a separate set of 416 predications used only for model selection.
Limitations
- The model was trained on SemRep predications from PubMed abstracts and may not generalize to other text sources, relation extraction systems, or predicate types.
- The model assesses whether a predication is stated in its source sentence, not whether the statement is factually true.
Citation
@article{zhang2021drug,
title = {Drug repurposing for {COVID-19} via knowledge graph completion},
author = {Zhang, Rui and Hristovski, Dimitar and Schutte, Dalton and Kastrin, Andrej and Fiszman, Marcelo and Kilicoglu, Halil},
journal = {Journal of Biomedical Informatics},
volume = {115},
pages = {103696},
year = {2021},
doi = {10.1016/j.jbi.2021.103696}
}
- Downloads last month
- 66
Model tree for xincan/pubmedbert-semrep-accuracy-classifier
Paper for xincan/pubmedbert-semrep-accuracy-classifier
Evaluation results
- Test precision on SemRep predication annotations (held-out test set, 703 predications)test set self-reported0.871
- Test recall on SemRep predication annotations (held-out test set, 703 predications)test set self-reported0.815
- Test F1 on SemRep predication annotations (held-out test set, 703 predications)test set self-reported0.842