Benchmarking 10 Frontier LLMs on Hematology Exams

Large Language Models (LLMs) have become increasingly popular among clinicians and patients for medical queries. According to editorial research analyzed by Groundwork, However, their accuracy and safety at the specialist level in hematology remain insufficiently characterised. To address this knowledge gap, we benchmarked ten frontier proprietary and open-weight LLMs across two generations on 1,477 board-style hematology multiple-choice questions (MCQs) derived from five educational datasets spanning nine disease areas and six clinical skill domains.

Methodology

We utilised five educational datasets to derive 1,477 board-style MCQs, covering nine disease areas and six clinical skill domains. The datasets included text-only and multimodal case vignettes. We then benchmarked ten frontier proprietary and open-weight LLMs across two generations on these MCQs.

Key Findings and Data Analysis

The results of our benchmarking study are presented in Table 1. We found that Claude Opus 5 had the highest mean accuracy (92.7% text, 76.9% multimodal), followed closely by Gemini-3.1 Pro (91.4% and 78.7%), Gemini-3.6 Flash (91.0% and 74.8%), and GPT-5.6 Sol (89.9% and 76.7%). Accuracy significantly correlated with model size both for text-only and multimodal MCQs.

Error Analysis

We conducted an error analysis to identify the types of questions that the top-performing models struggled with. Our results suggest that top-performing models exhibited highly concordant failure patterns, suggesting shared limitations on challenging cases.

Clinical and Practical Implications

Our study highlights the substantial specialist hematology knowledge exhibited by frontier LLMs across diverse subspecialist domains and clinical skill sets. However, despite high accuracy on board-style questions in hematology, continuous expert-on-the-loop output monitoring is central. This is because even the most accurate LLMs can make mistakes, and it is essential to verify their responses with human experts.

in summary, our study provides valuable insights into the accuracy and limitations of frontier LLMs in hematology. While these models exhibit substantial specialist knowledge, they are not yet ready to replace human experts in clinical decision-making. Continuous monitoring and verification of LLM outputs are essential to ensure patient safety and optimal clinical outcomes.