RealmPathology Benchmark

Model scoreboard

We ran the dataset of expert authored pathology tasks against thirteen frontier models, three independent rollouts per task. Scores are the mean of task means: for each task the mean verifier reward over three rollouts, averaged across the dataset. Reward is the share of positive rubric weight a model earns, after penalties, floored at zero.

What changed in this release

This release retargets the benchmark. The dataset and the expert answers are unchanged, but every task was reframed and every rubric rebuilt around four clinical failure modes that clinicians report in real operational use of these models:

• stating a definitive diagnosis on insufficient evidence,
• recommending treatment that is inaccurate or not justified,
• failing to reconcile uncertainty and name the steps to verify a finding, and
• reading the report too literally, missing what the findings mean.

Two mechanical changes follow. Prompts now permit a model to declare what cannot be determined, ask for the steps needed to verify a finding where confirmation is required, and ask the model to weigh further testing or treatment against its actual benefit. Rubrics now carry 299 penalty criteria alongside 772 positive ones: a penalty fires when a summary invents a finding, misclassifies the case, or recommends beyond what the report supports, and it subtracts from the score. The earlier rubrics had no penalties, so these results are a new comparison identity and are not continuous with the earlier release.

Headline findings

Three findings shape how these results should be read. Each is stated plainly first, then unpacked in its own section below.

Benchmark overview

Each task drops a model into a fresh sandbox with one or more real anatomic pathology reports and a generic toolkit: a shell, file read, write and edit, search over files, a todo list, and web search and web fetch for guideline lookups the report does not contain. The agent extracts the report, typically with pdftotext or a Python library, analyzes it, and writes a single structured interpretation to response.md. An LLM judge then scores that answer against an expert authored rubric of weighted criteria, one call per criterion. Most criteria are positive, for example states the Grade Group as 4; a substantial set are penalties, for example recommends a specific procedure the report does not support, which subtract. The run reward is earned positive weight over total positive weight, floored at zero, so a summary that fabricates a result can score below what its correct content alone would earn.

Task taxonomy

The tasks are drawn from routine diagnostic pathology and span the organ systems and specimen types a working service sees. The mix is intentional: most tasks are single report interpretations, but a meaningful fraction are multi report cases, marrow workups that combine myelogram, flow cytometry, biopsy and cytogenetics, which require reconciling several modalities into one read. The corpus is unchanged from the previous release; the retarget is in the prompts and rubrics, not the reports.

Share of the dataset by category. Groupings are approximate; some multi organ cases are filed under their dominant specimen. Roughly four in five tasks are single report interpretations; the remainder are multi report integrations.

Performance summary

Best of 3 takes the highest of three rollouts per task before averaging. Pass@3 is the probability at least one of three attempts scores 0.8 or more. Perfect is the share of runs at exactly 1.0. Tasks won is the share of the dataset on which the model has the single best per task mean; ties are not counted.

Paired differences

With thirteen models the full pairwise grid is large, so the table shows Claude Fable 5.1 against each other model; the clusters behind the leader are described below. Separated means the bootstrap interval over tasks excludes zero.

Fable separates from every model except Claude Sonnet 5.5, Claude Opus 5.5, Muse Wonder and Muse Spark 1.3. The top of the field is a chain rather than a single leader: Sonnet 5.5 and Opus 5.5 are each not separated from Muse Wonder, Muse Spark 1.3, Claude Opus 5 or Muse Spark 1.1. In the middle, adjacent models from Grok 4.7 to GPT-6 Sol are within noise of each other, though the ends of that group separate. Gemini 3.8 Flash is last and separated from every model except GPT-6 Sol. Won, lost and tied are shares of the dataset.

Penalties that fired most, with the judge's reason

The penalties one model tripped on every run while most other models avoided them. Each card shows the behavior the rubric forbids, how often that model and the rest of the field did it, and the judge's reason, with the words quoted from the model's summary highlighted.

Where the points go

Every positive criterion was classified by the kind of work it tests: Extraction restates a finding written in the report, Classification reaches the right diagnosis, grade, stage or risk group under a named system, Interpretation explains what the findings mean and their limits, Management recommends the next step, and Communication completes the form, states uncertainty and stays in scope. The classification is not cosmetic: it shows where the points are lost, and it is how the retarget is enforced, since the weight the failure modes concern sits in interpretation, management and the uncertainty facing part of communication.

Weighted pass rate on positive criteria, by type, per model, over all runs. Criteria per type: Extraction 328 (36% of positive weight), Classification 118 (18%), Interpretation 200 (30%), Management 93 (11%), Communication 33 (4%). Extraction is near saturated; Interpretation has the lowest pass rate and, carrying 30% of the weight, is where the most reward is lost.

Report knowledge versus guideline knowledge

The gap widens when a criterion needs knowledge the report does not print, such as a staging table or a guideline threshold, rather than a fact stated on the page.

Weighted pass rate on positive criteria, split by whether satisfying the criterion needs knowledge not printed in the report and by whether it hinges on a specific number.

What a miss looks like

One failed criterion per model and type, chosen where that model missed what the other models found, with the judge's reason in its own words. Pick a criterion type.

Hardest tasks: where every model struggled

These are the lowest reward tasks in the benchmark. The pattern is consistent with the second finding: each one punishes saying more than the specimen supports. The anchor column is the criterion most responsible for lost reward, read it as the thing the report did not let you say.

Lowest per task mean reward across models, three rollouts each.

Largest model spreads: where behavior separated most

These tasks produced the widest gap between the strongest and weakest model on the same problem, the most informative cases for understanding what distinguishes a good pathology read from a weak one.

Highest per task spread between the best and weakest model, excluding tasks already listed among the hardest. Several of these are the cases the failure mode examples below draw on.

Where models lose reward

The four failure modes that define the retarget are below, each with a complete example: the task, the exact rubric criterion, the model response, the judge verdict and the judge justification, so the failure is legible rather than asserted. No model is immune, and the examples below are drawn from the top of the field, where the errors are subtle enough to survive an otherwise strong summary.

!Definitive answer where the report withholds one

The model states a diagnosis, a stage, or a value the report does not support, in place of holding the category the report actually renders. This is the overreach that dominates the observed penalties, and it is a discipline problem more than a knowledge problem.

The correct answer holds the reported category; overriding it with a more confident reclassification invents a call the specimen does not carry. This is overreach, the largest penalty category in the set: 123 of the 299 penalty criteria describe it, against 105 for misclassification and 53 for fabrication.

→Inaccurate or unjustified management

The task asks for a pathology report interpretation, and the model answers like a treating clinician, naming a test or therapy the report records only as a suggestion, or gives a recommendation that is close to right but not the one this case calls for. Overreach into management is the single most tripped penalty; the mirror problem is a correct recommendation stated too loosely to earn its criterion.

The report records a suggestion; asserting that germline testing is indicated invents a downstream decision the report does not make. The retargeted rubrics require the benefit of any recommended testing to be stated, not only its justification, so a recommendation that is reasonable in general but not the one this report calls for does not earn the criterion.

?Uncertainty resolved instead of verified

The report deliberately leaves a question open, an unresolved differential, a discordance between modalities, a result that must be confirmed, and the model either resolves it too decisively or acts on it without naming the step that would verify it.

Acting on a result is not enough; the focus requires naming the confirmation the result still needs. The same pattern runs through the hardest cases: on report_2605222 the rubric asks the model to state that the HPV 52 versus HPV 18 ranking is population dependent rather than universal, a criterion no model met in any run, and on report_2605281 to hold management open until the pending HPV PCR result is incorporated rather than finalizing on histology alone.

≡Reading the report too literally

The model restates the report faithfully but does not read it for meaning: it repeats a staging code without saying what the code implies, applies a named system by rote to a case it does not fit, or relabels a specimen more specifically than the report allows.

The report states a code; the model must state its meaning. Over literal reading also drives the interpretation misses that separate the field: on report_2605191 the ascending colon tumor must be reported as a conventional adenocarcinoma rather than relabeled as a serrated subtype, and on report_2605291 the rubric asks the model to note that the infiltrative follicular subtype carries a less favorable behavior than the classic subtype, a reading no model made in any run.

Three sample tasks, every model

Three single report tasks from the benchmark, shown in full. For each model the median of its three rollouts is shown: the graded criteria with the judge's reasoning, the interpretation as written, and the trajectory that produced it. Pick a task, then a model.

Reading effort, not just score

Effort diverges far more than score. The table below is the mean per task effort of each model. The models agree on which tasks are hard, the rank correlation of per task means between any two models is 0.57 to 0.87, and repeats are stable, with a within task standard deviation of 0.045 for GPT-6 Astra to 0.067 for Claude Sonnet 5. What they do not share is how much work they spend reaching the answer.

Mean per task effort. Claude Opus 5 writes 2,525 words to GPT-6 Sol 456; a run costs about 0.13 dollars on Muse Spark 1.1 and 2.95 dollars on Claude Opus 5.5 (GPT-6 Sol was not yet priced on the platform when it ran). Summary length barely tracks reward: the per model correlation between words and score runs from +0.18 to -0.22.

Length is not care. Claude Opus 5.5 is the clearest case: it looks things up on the web in 87% of runs, delegates to subagents, and spends 19.2 minutes and 32.4 tool calls per task, yet finishes within noise of Claude Fable 5.1 and below it. Claude Sonnet 5.5, the next heaviest, reasons the most of any model, 77.9k tokens per task, and finishes second, also within noise of the leader; Muse Wonder spends 2.1 minutes per task and finishes fourth. Across the field, the heaviest runs were frequently among the weakest, and word count barely tracks reward at all. On report_2605240 the run that reclassifies a benign call to nondiagnostic loses a criterion the leaders hold; on report_2605258 even the strongest models score below 0.5, because the whole task turns on declaring an unknown rather than producing a number. The strongest reads were short and disciplined: extract, state the limit, and stop.

What this means for deploying models on pathology workflows

RealmPathology Benchmark is hard because it is concrete: the rubric grades exact figures, preserved limits, and the absence of unsupported escalation, with no credit for sounding authoritative. Three takeaways follow.

1. Treat extraction as reliable and put human review where judgment happens.All thirteen models read messy reports and produce coherent structured summaries; extraction criteria are met 93% of the time. The reward is lost downstream, at the point where the model decides whether to stop at the report. Human review is most valuable there, not on the extraction layer.
2. The dangerous errors are confident additions and resolved uncertainty, not missing facts.Overreach is the largest penalty category, 123 of the 299, and the penalties reorder the field: on positive criteria alone the order would be Fable, Opus 5, Opus 5.5, Sonnet 5.5, Wonder, Muse 1.1, Muse 1.3, Sonnet 5, Gemini, Grok 4.7, Grok 4.6, Sol, Astra. A reviewer should check that every stage, biomarker and recommendation in the output is present in the source report, and that every stated certainty is one the report actually supports.
3. Score behavior, and do not reward effort.The models cluster tightly on headline score, and the separations that exist are penalty separations: Gemini 3.8 Flash gives back 0.159 of its earned weight to penalties while GPT-6 Astra gives back only 0.044. The most process heavy agent in the field, Claude Opus 5.5, costs more than twice as much per run as the leader without beating it. Teams evaluating models for this work should read how an answer was produced and treat length as a warning sign, not a proxy for care.