BiomedParse

A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities

Theodore Zhao*¶1, Yu Gu*1, Jianwei Yang1, Naoto Usuyama1, Ho Hin Lee1, Sid Kiblawi1, Tristan Naumann1, Jianfeng Gao1, Angela Crabtree2, Jacob Abel2, Christine Moung-Wen2, Brian Piening2,3, Carlo Bifulco2,3, Mu Wei‡1, Hoifung Poon‡§1, Sheng Wang‡4,5
1Microsoft Research, 2Providence Genomics, 3Earle A. Chiles Research Institute, Providence Cancer Institute, 4Paul G. Allen School of Computer Science and Engineering, University of Washington, 5Department of Surgery, University of Washington
* Equal contribution Main technical contribution ‡ Corresponding authors § Lead contact
arXiv Paper Code Data
Retinal OCT Retinal OCT with Edema Fundus Retinal Optic Disc in Retinal Fundus Brain MRI-FLAIR Lower-Grade Glioma in Brain MRI-FLAIR Retinal OCT Retinal OCT with Edema Fundus Retinal Optic Disc in Retinal Fundus Brain MRI-FLAIR Lower-Grade Glioma in Brain MRI-FLAIR
CT Lung COVID-19 Infection in Lung CT Chest X-Ray COVID-19 Infection in Chest X-Ray Colon Endoscope Neoplastic Polyp in Colon Endoscope CT Lung COVID-19 Infection in Lung CT Chest X-Ray COVID-19 Infection in Chest X-Ray Colon Endoscope Neoplastic Polyp in Colon Endoscope
Skin Dermoscopy Melanoma in Skin Dermoscope Ultrasound Breast Malignant Tumor in Breast Ultrasound Pathology Colon Glandular Structure in Colon Pathology Skin Dermoscopy Melanoma in Skin Dermoscope Ultrasound Breast Malignant Tumor in Breast Ultrasound Pathology Colon Glandular Structure in Colon Pathology

Segmentation

BiomedParse performs segmentation for organs, abnormalities and cells, accurately following user's prompts. Without any image specific guidance like bounding box or points, BiomedParse outperforms state-of-the-art bounding box methods with text prompts only, across 9 biomedical imaging modalities.

BiomedParse Demo

Detection

BiomedParse detects the specific object of interest, and locate it at pixel-level precision, even for objects with irregular shapes. By effectively identifying text prompts describing object that does not exist in the image, BiomedParse is capable of object detection in an end-to-end manner.

Advanced Detection Demo

Recognition

Tired of typing prompts for every objects? BiomedParse can do object recognition all at once. Having learned 82 object types, BiomedParse can automatically identify all objects in a given image along with their semantic types, and simultaneously segment and label all biomedical objects of interests.

BiomedParse Demo

One model, 9 imaging modalities


"COVID-19 infection in chest CT"

"Glandular structure in colon Pathology"

"Neoplastic polyp in colon Endoscope"

"Lower-grade glioma in brain MRI"

"Malignant tumor in breast Ultrasound"

"Melanoma in skin Dermoscopy"

"COVID-19 infection in chest X-Ray"

"Optic disc in retinal Fundus"

"Cystoid macular edema in retinal OCT"

Abstract

Biomedical image analysis is fundamental for biomedical discovery. Holistic image analysis comprises interdependent subtasks such as segmentation, detection and recognition, which are tackled separately by traditional approaches. Here, we propose BiomedParse, a biomedical foundation model that can jointly conduct segmentation, detection and recognition across nine imaging modalities. This joint learning improves the accuracy for individual tasks and enables new applications such as segmenting all relevant objects in an image through a textual description. To train BiomedParse, we created a large dataset comprising over 6 million triples of image, segmentation mask and textual description by leveraging natural language labels or descriptions accompanying existing datasets. We showed that BiomedParse outperformed existing methods on image segmentation across nine imaging modalities, with larger improvement on objects with irregular shapes. We further showed that BiomedParse can simultaneously segment and label all objects in an image. In summary, BiomedParse is an all-in-one tool for biomedical image analysis on all major image modalities, paving the path for efficient and accurate image-based biomedical discovery.