Realm tests one job: read a real anatomic pathology report and produce the structured interpretation a tumor board can act on. Reading the report is the easy part. Success is defined by what the model does once it has read the report correctly. It must reach a diagnosis, stage or risk group only when the specimen supports one, reconcile the uncertainties the report leaves open rather than resolving them by assertion, recommend only the management the report justifies, and read each finding for its clinical meaning rather than its literal text. Every finding below ties back to those axes.

We ran the dataset of expert authored pathology tasks against eleven frontier models, three independent rollouts per task. Scores are the mean of task means: for each task the mean verifier reward over three rollouts, averaged across the dataset. Reward is the share of positive rubric weight a model earns, after penalties, floored at zero.

This release retargets the benchmark. The dataset and the expert answers are unchanged, but every task was reframed and every rubric rebuilt around four clinical failure modes that clinicians report in real operational use of these models:

• stating a definitive diagnosis on insufficient evidence,
• Recommending treatment that is inaccurate or not justified,
• Failing to reconcile uncertainty and name the steps to verify a finding, and
• Reading the report too literally, missing what the findings mean.

Two mechanical changes follow. Prompts now permit a model to declare what cannot be determined, ask for the steps needed to verify a finding where confirmation is required, and ask the model to weigh further testing or treatment against its actual benefit. Rubrics now carry 299 penalty criteria alongside 772 positive ones: a penalty fires when a summary invents a finding, misclassifies the case, or recommends beyond what the report supports, and it subtracts from the score. The earlier rubrics had no penalties, so these results are a new comparison identity and are not continuous with the earlier release.

Three findings shape how these results should be read. Each is stated plainly first, then unpacked in its own section below.

Each task drops a model into a fresh sandbox with one or more real anatomic pathology reports and a generic toolkit: a shell, file read, write and edit, search over files, a todo list, and web search and web fetch for guideline lookups the report does not contain. The agent extracts the report, typically with pdftotext or a Python library, analyzes it, and writes a single structured interpretation to response.md. An LLM judge, then scores that answer against an expert authored rubric of weighted criteria, one call per criterion. Most criteria are positive, for example states the Grade Group as 4; a substantial set are penalties, for example recommends a specific procedure the report does not support, which subtract. The run reward is earned positive weight over total positive weight, floored at zero, so a summary that fabricates a result can score below what its correct content alone would earn.

The tasks are drawn from routine diagnostic pathology and span the organ systems and specimen types a working service sees. The mix is intentional: most tasks are single report interpretations, but a meaningful fraction are multi report cases, marrow workups that combine myelogram, flow cytometry, biopsy and cytogenetics, which require reconciling several modalities into one read. The corpus is unchanged from the previous release; the retarget is in the prompts and rubrics, not the reports.

With eleven models the full pairwise grid is large, so the table shows Claude Fable 5.1 against each other model; the clusters behind the leader are described below. Separated means the bootstrap interval over tasks excludes zero.

Every positive criterion was classified by the kind of work it tests: Extraction restates a finding written in the report, Classification reaches the right diagnosis, grade, stage or risk group under a named system, Interpretation explains what the findings mean and their limits, Management recommends the next step, and Communication completes the form, states uncertainty and stays in scope. The classification is not cosmetic: it shows where the points are lost, and it is how the retarget is enforced, since the weight the failure modes concern sits in interpretation, management and the uncertainty facing part of communication.

The gap widens when a criterion needs knowledge the report does not print, such as a staging table or a guideline threshold, rather than a fact stated on the page.

These are the lowest reward tasks in the benchmark. The pattern is consistent with the second finding: each one punishes saying more than the specimen supports. The anchor column is the criterion most responsible for lost reward, read it as the thing the report did not let you say.

These tasks produced the widest gap between the strongest and weakest model on the same problem, the most informative cases for understanding what distinguishes a good pathology read from a weak one.

The four failure modes that define the retarget are below, each with a complete example: the task, the exact rubric criterion, the model response, the judge verdict and the judge justification, so the failure is legible rather than asserted. No model is immune, and the examples below are drawn from the top of the field, where the errors are subtle enough to survive an otherwise strong summary.

Effort diverges far more than score. The table below is the mean per task effort of each model. The models agree on which tasks are hard, the rank correlation of per task means between any two models is 0.57 to 0.87, and repeats are stable, with a within task standard deviation of 0.045 for GPT-6 Astra to 0.067 for Claude Sonnet 5. What they do not share is how much work they spend reaching the answer.

Realm is hard because it is concrete: the rubric grades exact figures, preserved limits, and the absence of unsupported escalation, with no credit for sounding authoritative. Three takeaways follow.

The dataset of expert authored pathology tasks, most single report interpretations and a smaller set of multi report integrations, was reauthored and signed off by domain reviewers over 17 to 20 September 2026. Eleven frontier models: Claude Fable 5.1 (max), Claude Opus 5.5 (max), Muse Spark 1.3 (xhigh), Claude Opus 5 (max), Muse Spark 1.1 (xhigh), Grok 4.7 (xhigh), Grok 4.6 (high), Claude Sonnet 5 (default), GPT-6 Astra and GPT-6 Sol (max, with the subagent task tool denied), and Gemini 3.8 Flash (high); three rollouts per task. The eight launch models ran in Daytona sandboxes; Grok 4.7, GPT-6 Sol and Claude Opus 5.5, released the following week, ran on the identical pinned task versions in Modal sandboxes. Twelve repetitions that failed for infrastructure reasons, a provider stream timeout or a harness idle stall with no answer written, were rerun once with the same configuration and matched by task, never selected by score; there were no exclusions. Reports are anonymized. Agents run in a sandboxed OpenCode container with the report PDFs available and a generic toolkit of a shell, file read, write and edit, search over files, a todo list, and web search and web fetch; each writes a single structured interpretation, which the verifier grades. One judge, GPT-5.4 mini, snapshot of 17 March 2026 at medium effort, grades every criterion against a hidden expert answer, one call per criterion. Rubrics carry 1,071 criteria, 772 positive and 299 penalty (overreach 123, misclassification 105, fabrication 53, omission 10, other 8), 12 to 29 per task. Reward is earned positive weight over total positive weight, floored at zero; a penalty subtracts its weight when the behavior is present. Because the earlier rubrics carried no penalties, these results are a new comparison identity and are not continuous with the earlier release.