HealthAgentBench
Modalities (6): text, EHR, 2D, CT, MRI, WSI
Workflow scopes (4): diagnosis, specimen assessment, research, data management
HealthAgentBench evaluates frontier agents on realistic healthcare workflows spanning nine task categories with distinct environments and diverse data modalities, including 2D chest X-rays, 3D CT volumes, MRI examinations, gigapixel whole-slide pathology images, free-text clinical documents, and structured EHR data. Each task is executed in a terminal environment using real clinical artefacts with minimal instructions, and evaluated against hidden gold labels to determine task success.
Frontier agents ranked by task success rate across 75 tasks, with one scored attempt per task (450 evaluations across six agents). All runs use xhigh reasoning, model API access only, and disabled web tools.
| # ▲ | Agent ↕ | Success Rate ↕ | cost/task ↕ |
|---|---|---|---|
| 1 | Claude Code (Opus 5.5) | $4.7 | |
| 2 | Claude Code (Sonnet 5.5) | $1.8 | |
| 3 | Codex (GPT-6-astra) | $3.8 | |
| 4 | Codex (GPT-6-sol) | $1.6 | |
| 5 | Codex (GPT-5.6-sol) | $2.5 | |
| 6 | Claude Code (Opus 5) | $6.7 |
Score metric: Mean task success rate across 75 tasks. Cost/task is total recorded agent cost divided by 75; costs are API-equivalent estimates.
This interactive version of the paper figure situates HealthAgentBench among prior healthcare-agent benchmarks. The horizontal axis groups descriptive interaction settings: static QA, conversation context or simulation, and executable environments. The vertical axis counts raw-input modalities, and marker size encodes workflow scopes. These scopes summarize principal evaluated purposes and may overlap; horizontal position within a column carries no meaning.
Each column is a task category with its unique environment. Each cell encodes three signals at once: color is the per-category success rate, circle size is average wall-clock time, and the green $ tier is the mean run cost.
The current release groups 75 tasks into 9 categories, each with its own unique environment. Each card below shows the category, modality, access regime, and best reported success rate.
Review a chest X-ray Findings draft against the target study and longitudinal history, then write the corrected report section. Success requires a majority of judges to find no significant errors.
Interpret a chest CT volume and emit a per-volume yes/no label vector for the report-grounded abnormality findings. Success criteria: 1.0 accuracy.
Inspect a gigapixel pathology slide and predict the set of tumor tiles on a fixed analysis grid. Success criteria: ≥ 0.9 tile F1.
Inspect eight whole-slide H&E images and identify every slide containing malignant tumor cells. Success requires an exact match to the tumor-slide set, including empty sets.
Review three MRI examinations from the supplied axial, coronal, and sagittal arrays. Answer ACL-tear and meniscal-tear questions for each exam. All six answers must match the source labels.
Audit an EHR corpus containing injected errors and flag affected rows. Success requires error-cluster recall of 1.0 and row-level precision of at least 0.1.
Answer numerical clinical questions from raw MIMIC-IV tables, using 1,020 labeled examples to validate calculations. Five question-category tasks and one combined task require every answer correct.
Learn from labeled longitudinal EHR timelines and predict held-out clinical-event risks using only observable history. Success requires AUROC to match or exceed the reference baseline.
Predict overall survival time for five cancer cohorts using labeled training data. Success requires full prediction coverage and better C-index, censoring-adjusted error, and integrated Brier score than both Cox and RSF baselines.
Help expand HealthAgentBench with realistic clinical workflows, datasets, and evaluation protocols from your domain.
If you use HealthAgentBench, cite it with the following BibTeX entry.
@misc{liu2026healthagentbench,
title = {HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents},
author = {Liu, Qianchu and Zhang, Sheng and Qin, Guanghui and Valanarasu, Jeya Maria Jose and Huang, Xiaoke and Argaw, Peniel and Rokuss, Maximilian and Lu, Mingyu and Ossowski, Timothy and Chaves, Juan Manuel Zambrano and Wong, Cliff and Patel, Naisargi and Hasija, Yashna and Wei, Mu and Yim, Wen-wai and Liu, Qin and Jing, Zilin and Entenmann, Jason and Usuyama, Naoto and Naumann, Tristan and Poon, Hoifung},
year = {2026},
eprint = {2606.31179},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2606.31179}
}