News

We will provide updates here for the BLURB benchmark and leaderboard.

Nov 1, 2023 - PubMedBERT renamed PubMedBERT has been renamed MSR BiomedBERT. This name change is now reflected in the leaderboard and links to models.

July 22, 2021 — Score correction
We found a small bug in our evaluation script for GAD, which resulted in slightly lower scores for our models as reported before. We have updated the paper and the leaderboard with the correct results.

August 5, 2020 — Model Release!
We’re pleased to announce the release of two pretrained PubMedBERT models, which are immediately available for download. These models represent the initial state of the art in terms of BLURB score. PubMedBERT (abstracts only) refers to the primary PubMedBERT model evaluated in our publication and details for the PubMedBERT (abstracts + full text) model can be found in section 6.4 of the same publication.

• PubMedBERT (abstracts only)
• PubMedBERT (abstracts + full text)

August 1, 2020 — Initial Release!
The initial benchmark contains the following data sets:

1. BC2GM (NER)
2. BC5-Chem (NER)
3. BC5-Disease (NER)
4. JNLPBA (NER)
5. NCBI-Disease (NER)
6. EBM PICO (PICO)
7. ChemProt (Relation Extraction)
8. DDI (Relation Extraction)
9. GAD (Relation Extraction)
10. BIOSSES (Sentence Similarity)
11. HoC (Document Classification)
12. BioASQ (Question Answering)
13. Pubmed QA (Question Answering)