Token Classification
Transformers
Safetensors
modernbert
medical
clinical
pii
de-identification
anonymization
privacy
multilingual
mmbert
openmed
Eval Results (legacy)

OpenMed

OpenMed mmBERT PII Multilingual Base v1

OpenMed on Hugging Face OpenMed SDK on GitHub pip install openmed License Apache-2.0

OpenMed's open token classifier for masking personal information in clinical text. It tags the identifiers in Spanish clinical notes (names, record and license numbers, dates, ages, addresses, phone numbers, e-mails) and returns each one as an exact character span with a score, ready to mask. It also covers English, French, German, Italian, Dutch and Swedish, tested on synthetic finance documents. Plain transformers, no extra library. Trained and evaluated by OpenMed. Apache-2.0, 307.5M parameters, runs on CPU.

Developed by OpenMed
Task PII masking, token classification with one generic label (B-PII, I-PII, O)
Output (start, end, score); text[start:end] is the exact span
Parameters 307.5M
Default threshold 0.35
Languages Spanish (clinical text); English, French, German, Italian, Dutch, Swedish (tested on synthetic finance text)
License Apache-2.0

About OpenMed

OpenMed builds open models and open-source tools for clinical and biomedical text that run on your own hardware.

Quick start

pip install -U torch "transformers>=5"  # tested with transformers 5.23.0
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "OpenMed/mmBERT-PII-Multilingual-Base-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()


def find_pii(text, threshold=0.35):
    """Spans to mask in one passage, as (start, end, score)."""
    enc = tokenizer(text, return_offsets_mapping=True, return_tensors="pt")
    offsets = enc.pop("offset_mapping")[0].tolist()
    with torch.inference_mode():
        probs = model(**enc).logits[0].softmax(-1).tolist()
    spans, current = [], []
    for (a, b), p in zip(offsets, probs):
        while a < b and text[a].isspace():
            a += 1
        while a < b and text[b - 1].isspace():
            b -= 1
        if a == b:  # special or whitespace-only token
            continue
        tag = max(range(3), key=p.__getitem__)  # 0 = O, 1 = B-PII, 2 = I-PII
        if tag != 2 and current:
            spans.append(current)
            current = []
        if tag:
            current.append((a, b, p[tag]))
    if current:
        spans.append(current)
    found = [(s[0][0], s[-1][1], sum(t[2] for t in s) / len(s)) for s in spans]
    return [(a, b, score) for a, b, score in found if score >= threshold]


text = (
    "Informe firmado por la Dra. Elena Sanz Prieto, número "
    "de colegiado 284512706, Servicio de Neumología. "
    "Correo: esanz.neumo@hospitalriotajo.es."
)
spans = find_pii(text)
for start, end, score in spans:
    print(start, end, text[start:end], f"{score:.2f}")

masked = text
for start, end, _ in reversed(spans):
    masked = masked[:start] + "[PII]" + masked[end:]
print(masked)
28 45 Elena Sanz Prieto 1.00
67 76 284512706 0.98
110 140 esanz.neumo@hospitalriotajo.es 1.00
Informe firmado por la Dra. [PII], número de colegiado [PII], Servicio de Neumología. Correo: [PII].

The settings

  • find_pii is the evaluation's decoding: the most likely tag per token (O, B-PII, I-PII), special and whitespace-only tokens skipped, a span from each B-PII (or I-PII after O) through the I-PII tokens that follow, scored by the mean of its tokens' probabilities, kept at 0.35 or more. Every span is one generic kind, personal information to mask.
  • threshold=0.35 was chosen on the tuning data to favor recall. Raise it for fewer, surer spans, at the cost of leaving more identifiers unmasked.
  • Pass one passage of up to about 400 characters: the model was trained and evaluated on windows of at most 438 UTF-8 bytes. For longer notes, see below.
  • The OpenMed SDK's analyze_text also runs this model, but it does not return the evaluated spans: on the ten examples below they match on 5 with the default config and on 6 with use_medical_tokenizer=False. The default joins neighboring spans (46002 Valencia); without the remap it splits spans at spaces the model tags as outside ((617) and 555-0142, Via Garibaldi 8, and 20121 Milano); both drop the punctuation-only span in example 1 and return 1958 or 958 where the evaluated decoding returns 9 in example 6. So this card uses plain transformers.

Long notes, many notes, GPU

For a note with several lines, run find_pii one line at a time and shift the offsets (keep each line under about 400 characters):

def find_pii_in_note(note, threshold=0.35):
    """Run find_pii line by line and return offsets into the whole note."""
    spans, offset = [], 0
    for line in note.split("\n"):
        spans += [(offset + a, offset + b, s) for a, b, s in find_pii(line, threshold)]
        offset += len(line) + 1
    return spans
  • The model loads once, so call find_pii in a loop for many notes.
  • On a GPU, add model.to("cuda") after loading and enc = enc.to("cuda") before the forward pass.

Evaluation

Spanish clinical case reports (main result): 91 case reports from the MEDDOCAN corpus, from 90 article groups held out of training and tuning, with 2,065 annotated identifiers. 946 of them are critical: names, ID numbers, e-mails, phone and fax numbers, relatives and street addresses. The case text is authentic; the identifiers in it were substituted or added by the corpus authors. All cases come from the MEDDOCAN authors' training file (the official test set was not used), so these numbers are not comparable to published MEDDOCAN results. Scores are counted per character in the original text: F2 weighs recall twice as much as precision, because a missed identifier costs more than an extra mask. An identifier is fully masked when every one of its characters is.

Three more views, on the same frozen model: 588 synthetic finance documents in seven languages (English, French, German, Italian, Spanish, Dutch and Swedish, 84 each) from 12 document types held out of training and tuning, where dates, times and company names are outside the mask scope (macro F2 is the mean of the seven languages); 107 person names in German court decisions, counted as fully masked or not (the gold marks names only, so there is no precision); and 24 short clinical notes with no personal data (12 Spanish, 6 German, 6 French), written as controls, counted as masked if anything in them is masked.

Model Spanish character F2 Spanish exact-span F1 Spanish critical identifiers fully masked Finance macro F2 (7 languages) German names fully masked Clean notes masked (of 24)
OpenMed mmBERT PII Multilingual Base v1 (this model), threshold 0.35 98.12 95.97 97.89 74.04 36.45 0
Earlier OpenMed PII model (EmbeddingGemma2 PII Multilingual 270M v2, frozen reference) 84.64 43.12 81.71 65.27 63.55 1
jhu-clsp/mmBERT-base with a new, untrained classification head (a floor, not a baseline) 38.46 2.96 28.22 19.17 30.84 24

Against the earlier trained reference, OpenMed's model gained +13.48 points of Spanish character F2 (84.64 to 98.12; 95% CI +12.52 to +14.51, paired bootstrap over the 90 article groups, 2,000 resamples), and the share of critical identifiers fully masked rose by +16.17 points (95% CI +14.02 to +18.30). The last row is the base encoder with a freshly initialized head, before any training, so it shows a floor rather than a competing model. The reference was chosen on the tuning data from three earlier OpenMed PII models, the threshold (0.35) was chosen on the tuning data too, and these cases were scored once, with the model, threshold and scoring code frozen first. The model was trained on 303 other MEDDOCAN case reports and 2,730 synthetic finance documents.

What it still misses: 28 of the 91 Spanish case reports kept at least one identifier character unmasked, and 13 kept part of a critical identifier (the reference: 91 and 82). It masked 0.09% of the characters that are not identifiers (the reference: 0.27%).

By identifier type (Spanish case reports)

Identifier (MEDDOCAN type) Critical Annotated Fully masked Share fully masked
place (city, province, postcode) (TERRITORIO) 332 327 98.49
date (FECHAS) 229 229 100.00
patient age (EDAD_SUJETO_ASISTENCIA) 189 183 96.83
patient name (NOMBRE_SUJETO_ASISTENCIA) yes 185 185 100.00
clinician name (NOMBRE_PERSONAL_SANITARIO) yes 182 182 100.00
patient sex (SEXO_SUJETO_ASISTENCIA) 172 170 98.84
street address (CALLE) yes 156 152 97.44
country (PAIS) 131 131 100.00
patient ID or record number (ID_SUJETO_ASISTENCIA) yes 105 98 93.33
e-mail (CORREO_ELECTRONICO) yes 85 85 100.00
clinician license number (ID_TITULACION_PERSONAL_SANITARIO) yes 84 84 100.00
insurance number (ID_ASEGURAMIENTO) yes 71 71 100.00
relatives (FAMILIARES_SUJETO_ASISTENCIA) yes 44 35 79.55
hospital (HOSPITAL) 42 41 97.62
encounter ID (ID_CONTACTO_ASISTENCIAL) yes 18 18 100.00
institution (INSTITUCION) 17 9 52.94
phone number (NUMERO_TELEFONO) yes 11 11 100.00
profession (PROFESION) 6 0 0.00
fax number (NUMERO_FAX) yes 5 5 100.00
other patient detail (OTROS_SUJETO_ASISTENCIA) 1 0 0.00

The weakest types (at least five annotations, under 95% fully masked): profession, 0 of 6; institution, 9 of 17; relatives, 35 of 44; patient ID or record number, 98 of 105.

Synthetic finance documents by language

Character F2 English French German Italian Spanish Dutch Swedish
OpenMed mmBERT PII Multilingual Base v1 89.27 75.96 78.81 76.78 68.72 59.56 69.20
Earlier trained reference 77.28 72.02 65.25 70.42 65.16 47.34 59.41
Untrained head 13.78 17.29 19.24 19.29 22.47 21.17 20.94

Dutch is its weakest finance language (59.56). Across the seven languages names are the most common finance identifier, and 670 of the 1,404 names (47.72%) were fully masked. These documents are synthetic, so the numbers show transfer across languages, not performance on real records.

German names are its weak spot: 36.45% of the 107 names fully masked (55.73% of name characters), against 63.55% for the earlier reference and 61.68% for OpenMed GLiNER2.5 PII Multilingual v1. For German text, prefer the GLiNER2.5 model.

The OpenMed multilingual PII models

Same training data, same frozen comparison, same views. GLiNER2.5 is ahead on every scored view, and neither masks any of the 24 clean notes; mmBERT runs with plain transformers and no extra library.

Model Parameters Default threshold Quick start Spanish character F2 Spanish critical identifiers fully masked Finance macro F2 (7 languages) German names fully masked Clean notes masked (of 24)
OpenMed GLiNER2.5 PII Multilingual v1 287.4M 0.15 OpenMed SDK 98.57 98.94 76.95 61.68 0
OpenMed mmBERT PII Multilingual Base v1 (this model) 307.5M 0.35 transformers 98.12 97.89 74.04 36.45 0
Earlier OpenMed PII model (EmbeddingGemma2 270M v2, reference, not released) 84.64 81.71 65.27 63.55 1

Examples

Real outputs of this model with the quick-start find_pii, with the default threshold 0.35, on CPU. The texts are synthetic, written for this card: every name, number, address, e-mail and hospital is invented. Spans to mask are in bold with their score.

  1. Admission note (Spanish): Paciente: Lucía Martín Herrera1.00, 58 años0.99, NHC 28471931.00. Ingresa el 14/03**/**0.522025 en el Hospital Universitario Río Tajo1.00 por disnea de esfuerzo.
  2. Signing physician (Spanish): Informe firmado por la Dra. Elena Sanz Prieto1.00, número de colegiado 2845127060.98, Servicio de Neumología. Correo: esanz.neumo@hospitalriotajo.es1.00.
  3. Address and phone (Spanish): Vive con su marido0.95 en la calle Pintor Sorolla 23, 4º B0.93, 460020.99 Valencia0.98. Teléfono de contacto: 961 234 5670.99.
  4. Relative and profession (Spanish): Varón1.00 de 71 años1.00, NASS0.58 28 41235678 090.99, que acude acompañado de su hija0.97 Carmen1.00, enfermera0.98 de profesión.
  5. English discharge (English): Discharge summary for John Patrick Miller0.99, DOB 04/17/19620.98, MRN 004829130.98. Follow-up with Dr. Sarah Chen0.94, phone (617) 555-01420.99.
  6. French referral (French): Mme Claire Dubois0.94, née le 3 mars 190.5458, domiciliée 12 rue des Lilas, 69003 Lyon0.97, est adressée par le Dr Antoine Mercier0.86.
  7. German bank details (German): Herr Klaus Becker0.92, geboren am 14.08.19710.86, wohnhaft in der Hauptstraße 27, 10115 Berlin, IBAN DE89 3704 0044 0532 0130 000.97.
  8. Italian tax code (Italian): Il signor Marco Bianchi0.88, codice fiscale BNCMRC80A01F205X, residente in Via Garibaldi 8, 20121 Milano0.98, telefono 02 4567 89100.96.
  9. Dutch account holder (Dutch): Rekeninghouder Jan de Vries1.00, Keizersgracht 1120.76, 10150.65 CW Amsterdam0.79, e-mail j.devries@voorbeeld.nl1.00.
  10. No personal data (Spanish): Se pauta amoxicilina 875 mg cada 8 horas durante 7 días y control de la tensión arterial en atención primaria. (nothing masked)

A longer Spanish note (803 characters, 17 spans) in one find_pii_in_note call:

Mujer de 46 años, natural de Toledo, que acude a urgencias del Hospital General Sierra Alta el 2 de febrero de 2025 por fiebre y dolor lumbar. Antecedentes: hipertensión arterial en tratamiento con enalapril. Vive en la avenida de la Constitución 51, 2º A, 45004 Toledo, con su pareja, Andrés Molina. NHC 5530918, NASS 45 10293847 22.
Exploración: temperatura 38,7 °C, puñopercusión renal derecha positiva. Analítica con leucocitosis y proteína C reactiva elevada. Se diagnostica pielonefritis aguda y se inicia ceftriaxona intravenosa.
Evolución favorable, con alta a las 72 horas y control por su médica de familia, la Dra. Beatriz Lozano Gil, en el centro de salud Los Almendros. Remitida por: Dr. Tomás Ibáñez Ruiz, nº de colegiado 454503218. Contacto: tibanez@hgsierraalta.es, teléfono 925 280 114.

Good to know

  • It masks relationship words and professions as well as names: marido (example 3), hija, Carmen and enfermera (example 4), pareja in the longer note. In the evaluation, though, professions were its weakest type (0 of 6 fully masked).
  • Dates and numbers can come back in pieces, leaving the rest visible: example 1 masks only one slash of 14/03/2025, example 6 only the 9 of 1958, example 9 only 015 of the postcode 1015 CW. It also masks ASS inside the word NASS (example 4).
  • It misses the German street address Hauptstraße 27, 10115 Berlin (example 7), in line with its weaker German results.
  • Titles are mostly left outside the span (Dr. before Sarah Chen, Herr before Klaus Becker; Mme stays inside Mme Claire Dubois), and in the longer note it masks Los Almendros but not centro de salud.
  • Example 10, a prescription with no personal data, comes back with nothing masked.
  • It misses the Italian tax code BNCMRC80A01F205X (example 8) and the written-out date 2 de febrero de 2025 in the longer note.

Limitations

  • The main result is in-distribution: the 91 Spanish case reports are MEDDOCAN article groups held out of the same author training file the model learned from, and their identifiers were substituted or added by the corpus authors. It was not tested on the official MEDDOCAN test set or on real hospital notes.
  • Clinical training data is Spanish only. The other languages were measured on synthetic finance documents, not on clinical notes, and Portuguese and other languages were not trained or tested.
  • German names: 36.45% fully masked, below the earlier reference's 63.55%; in the synthetic finance documents 670 of the 1,404 names were fully masked.
  • One generic label. It does not say which kind of identifier a span is, so type-aware replacement (fake names for names, shifted dates for dates) needs another step.
  • Scores rank spans; they are not calibrated probabilities.
  • Long texts are cut into pieces, and an identifier cut across two pieces can be partly missed.
  • It does not guarantee de-identification. Even on the in-distribution Spanish test, 28 of 91 case reports kept part of an identifier. Have a person review the output before any text leaves your control, and follow the rules that apply to your data.
  • The base model's pretraining data is not public.
  • It finds text spans. It does not give medical advice and is not a medical device.

Built on

OpenMed fine-tuned this model from jhu-clsp/mmBERT-base by the Johns Hopkins University Center for Language and Speech Processing (JHU CLSP), under MIT. Training and tuning data: Spanish clinical case reports from the MEDDOCAN corpus by Marimon et al. (2019, CC BY 4.0; Zenodo record 4279323), whose identifiers were substituted or added by the corpus authors, and the Gretel synthetic PII finance multilingual dataset (Apache-2.0). Evaluation only: person names from the German LER corpus of German court decisions by Leitner et al. (CC BY 4.0). OpenMed's changes: every identifier type was mapped to one generic mask, and MEDDOCAN cases linked to earlier OpenMed Spanish data (27) were left out. Thank you to JHU CLSP, to the MEDDOCAN authors, to Gretel and to the German LER authors for releasing their work openly.

This model is released under Apache-2.0. No corpus text is included in this repository.

Citation

@misc{panahi2025openmedner,
  title={OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets},
  author={Maziyar Panahi},
  year={2025},
  eprint={2508.01630},
  archivePrefix={arXiv}
}

@inproceedings{marimon2019meddocan,
  title={Automatic De-identification of Medical Texts in Spanish: the {MEDDOCAN} Track, Corpus, Guidelines, Methods and Evaluation of Results},
  author={Marimon, Montserrat and Gonzalez-Agirre, Aitor and Intxaurrondo, Ander and Rodriguez, Heidy and Lopez Martin, Jose Antonio and Villegas, Marta and Krallinger, Martin},
  booktitle={Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019)},
  series={CEUR Workshop Proceedings},
  volume={2421},
  year={2019}
}

@inproceedings{leitner2020german,
  title={A Dataset of German Legal Documents for Named Entity Recognition},
  author={Leitner, Elena and Rehm, Georg and Moreno-Schneider, Julian},
  booktitle={Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020)},
  year={2020}
}

@misc{marone2025mmbert,
  title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},
  author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
  year={2025},
  eprint={2509.06888},
  archivePrefix={arXiv}
}

OpenMed · Hugging Face · GitHub · PyPI · X

Downloads last month
4
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for OpenMed/mmBERT-PII-Multilingual-Base-v1

Finetuned
(213)
this model

Datasets used to train OpenMed/mmBERT-PII-Multilingual-Base-v1

Papers for OpenMed/mmBERT-PII-Multilingual-Base-v1

Evaluation results

  • Character F2 (threshold 0.35) on MEDDOCAN (91 held-out Spanish clinical case reports)
    self-reported
    98.120
  • Exact-span F1 (threshold 0.35) on MEDDOCAN (91 held-out Spanish clinical case reports)
    self-reported
    95.970
  • Critical identifiers fully masked (threshold 0.35) on MEDDOCAN (91 held-out Spanish clinical case reports)
    self-reported
    97.890
  • Character F2, macro over 7 languages (threshold 0.35) on Gretel synthetic PII finance multilingual (588 held-out documents, 7 languages)
    self-reported
    74.040
  • Names fully masked (threshold 0.35) on German LER (107 person names)
    self-reported
    36.450