macrotome SegFormer fast

SegFormer with the small MiT-B0 encoder (3.7 M parameters) and three sigmoid heads, trained with the same recipe and labels as macrotome SegFormer (MiT-B2). Made for computers without a GPU: about 30 s per 50 × 25 mm slide on 24 CPU cores (SegFormer 70 s), 1.5 s on an RTX 3090 (SegFormer 2 s), in 2048 px tiles at 8 µm/px. On the larger test sets it has about 0.5 mm² more tissue error per slide than SegFormer; on the held-out set the two score alike.

Segmentation of brightfield whole-slide images into three layers at 8 µm/px, for any stain, organ and scanner:

  • tissue: every piece of tissue on the glass, including fat, pale IHC and small fragments;
  • pen: marker ink and printed marks, on glass or over tissue;
  • quality: image defects such as bubbles, folds, blur, debris, coverslip edges and dark scan corners.

Part of macrotome. Other macrotome models: SegFormer, SAM 2.1, SAM 3.

Use

pip install "macrotome[segformer] @ git+https://github.com/cancercenterai/macrotome"
macrotome segment slide.svs --model segformer-fast --out out/
import macrotome

model = macrotome.load("segformer-fast")   # downloads the weights from this repository
res = model.segment("slide.svs")
tissue, pen, quality = res.mask("tissue"), res.mask("pen"), res.mask("quality")
res.write("out/")                       # label TIFF, QuPath GeoJSON, JSON summary, overlay

QuPath

segformer_b0.torchscript.pt (float32, for the CPU and Apple's GPU) and segformer_b0.torchscript-bf16.pt (bfloat16, for an NVIDIA GPU, as the published scores) are this model as TorchScript, for the macrotome QuPath extension, which downloads them itself. They take float RGB in 0..255 and return the three probabilities; the tile size, overlap, the 32 px white margin at image edges and the resolution are in macrotome.json inside each file.

Results

98 held-out slides (Tissues with Issues, Conflux, pathologist-labelled breast slides). The SegFormers were trained on none of them. The two SAM models were trained on an earlier label snapshot that still held the 20 breast slides; on those 20 slides they score the same as SegFormer models that never saw them (Dice 0.967–0.971).

Tissue Dice median / p10 False tissue mm²/slide Pen F1 Quality P / R
SegFormer 0.985 / 0.954 3.7 0.938 0.71 / 0.79
SegFormer fast 0.984 / 0.953 3.7 0.932 0.75 / 0.74
SAM 2.1 tuned 0.984 / 0.951 5.3 0.903 0.83 / 0.65
SAM 3 tuned 0.984 / 0.958 3.9 0.932 0.87 / 0.59
SlideSegmenter 0.951 / 0.751 14.2 0.691 –

Training data, evaluation on the public test sets, examples and limitations are in the repository README.

Licence

CC BY-NC-SA 4.0: research and other non-commercial use, with attribution, and adaptations under the same terms. The training data include sets under non-commercial and share-alike terms. The encoder starts from NVIDIA's MiT-B0 ImageNet weights (NVIDIA Source Code License, non-commercial).

Not for clinical use: the models are research tools and are not validated for diagnosis or any clinical decision.

Downloads last month
20
Safetensors
Model size
3.72M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for CancerCenterAI/macrotome-segformer-fast

Base model

nvidia/mit-b0
Finetuned
(431)
this model