%20(1).webp)
Key findings
1
Gemini 4 Argon and Claude Fable 5.1 are the only models to cross 50% on LegalReasonBench, and no model cleared every criterion on any one matter.
Argon scores 50.8% against 50.6% for Fable and 46.5% for Claude Opus 5.5, and Argon wins more matters outright than any other model. Argon reaches its score at $6.71 of solver cost per matter, about a third of Fable’s $19.74, though it takes longer to get there, 45 minutes per matter at the median against Fable’s 38. Under the all-pass standard that Harvey LAB uses, every model would score zero on LegalReasonBench.
2
Every model spots legal issues better than it identifies governing law or applies it to the facts.
Measured on the IRAC framework, every model scores higher on Issue than on Rule and Application, by 2 to 20 percentage points and 7 to 25 points respectively, and those two components carry 80% of the benchmark’s positive weight. Models find the legal neighborhood and then miss the controlling authority or misapply it, even when the record supplies the law.
3
Legal research failures compound legal reasoning failures.
LegalReasonBench splits its matters between a closed universe, where the record supplies the governing law, and an open universe, where the model must find it through its own research. Moving from closed to open, every model’s Rule score sees a fall of 26 to 38 percentage points and its Application score a fall of 11 to 24. The models that search the most score no better than the models that search the least, which points to the quality of the research rather than the amount.
4
The highest-scoring models write the most, but length alone does not predict the ranking.
Doubling the length of a deliverable adds about 3 points on average once the assignment and model are held fixed, though Gemini 4 Argon gains 18 and Claude Fable 5.1 gains 9. LegalReasonBench caps court-facing filings at a court’s word limits and leaves memoranda uncapped. Most models respect the limits without enforcement, and Fable is the least compliant at 56%.
5
A strong start on a long-horizon matter does not predict a strong finish.
A model’s performance on the first assignment barely predicts its performance on the later ones, and the pattern holds whether or not the later assignments depend on the first and whether or not the record changes between them.
1. Introduction
Why we built LegalReasonBench
LegalReasonBench is the first benchmark to measure legal reasoning on realistic, expert-authored, long-horizon legal workflows. Legal reasoning is the shared judgment that defines elite lawyering. Lawyers may disagree about exactly what a strong work product should look like, but they agree much more on the reasoning required to produce one. They identify the issues, determine the governing rules, apply those rules to the record, and reach supported conclusions. U.S.-educated attorneys know that structure as IRAC, and that consensus gives LegalReasonBench a basis for decomposing legal reasoning into measurable parts.
LegalReasonBench is designed to maximize realism. Existing benchmarks built on bar-exam style prompts, such as LegalBench, use a format that frontier models have since saturated, and benchmarks built on synthetic tasks and materials, such as Harvey LAB, leave out the messiness of real legal work. LegalReasonBench uses matters that practicing lawyers build end to end, from sourcing the matter and its underlying files to writing the prompts and the evaluation rubrics, and that run from early research through filing-ready advocacy and decision-ready analysis. The client facts and procedural details stay in the uploaded case files rather than being paraphrased into the prompt, so the model has to reason where the facts are messy.
LegalReasonBench’s rubrics are designed for diagnostic resolution. An all-pass standard for task completion, which Harvey LAB uses, would score every model evaluated here at 0%. LegalReasonBench instead scores weighted criteria, each mapped to an IRAC component, which preserves the signal about the individual legal judgments a model gets right or wrong. That signal can guide future evaluation, research, and post-training.
What this update adds
We published preliminary LegalReasonBench results in May 2026. This update changes the agent’s environment, the benchmark’s task composition, and the IRAC classification.
The agent’s environment is now a true multi-step workflow that mimics how a long-horizon legal matter unfolds. Each matter runs as a sequence of two to five assignments in one persistent sandbox, and each assignment sees only what a lawyer would know at that stage. Because each assignment is scored on its own, we can follow a model’s trajectory across the matter and see how its performance changes from one assignment to the next.
The task composition is more realistic. In May, every task was open universe with web search on. The updated LegalReasonBench splits evenly between open and closed universe, and the closed matters confine the model to an uploaded record that supplies the governing law along with the facts, so we can separate legal reasoning from legal research. Court-facing work product is now subject to a filing word limit. The role prompt holds motions, pleadings, and briefs to the limits of the Southern District of New York, and memoranda, as internal work product, have none.
The IRAC classification is more robust. A three-judge panel now assigns each rubric criterion to its IRAC component, in place of the single classifier used in May.
2. Methodology
Task construction
The 50 matters were selected to represent realistic and demanding legal workflows, across practice area, deliverable type, research universe, and horizon structure. Each matter is a sequence of two to five assignments worked in one persistent sandbox, and a practicing lawyer authored the matter, its case file, and its rubrics end to end. The table below summarizes the task set along each of those dimensions.
The practice areas overlap with Harvey LAB v1.0, which is largely corporate in focus, and add four more: criminal law and procedure, civil rights, administrative law, and sports law. The additions broaden the benchmark beyond the work of large law firms.
Every matter comes with at least three case files, the documents a lawyer would have on the desk, from pleadings and orders to deposition excerpts and exhibits. The client facts and procedural details stay in those files rather than being summarized into the prompt. Practicing lawyers authored and curated the files. Most matters carry a modest record, with a third holding five or six files and a further fifth seven to nine, but nearly a third carry ten or more, and a handful run past fifteen. Measured in extractable text, half the matters stay under 50,000 tokens, and the rest spread across a long tail, with one matter in six carrying 400,000 tokens or more
Agent’s environment
Each matter drops the model into a sandboxed container with the client's case file under /app/attachments/ and, where the matter allows research, an internet connection. Assignments arrive one at a time. The model writes the deliverable for the current assignment to a fixed path, the verifier grades that file, and the next assignment is issued into the same sandbox, where the earlier documents and the earlier deliverable are still on disk.
Later assignments introduce new facts, new filings, a new jurisdiction, or a new posture. In 68% of the matters, the later work materially requires understanding or building on the first assignment. In 86% of the matters, the factual, documentary, procedural, or legal record changes after the first assignment. These two dimensions, dependency and record evolution, let us test whether a model carries its earlier reasoning forward and whether it revises that reasoning when the record moves.
Evaluation protocol
Run settings
The harness is OpenCode 1.18.31, with reasoning effort set to high for every model. The seven models of the controlled rerun ran on Modal sandboxes with the subagent (Task) tool removed. Gemini 4 Argon* ran on Harbor-hosted sandboxes, with three of its matters rerun on Daytona sandboxes; its Task tool was not removed, but it never called it, so every model works alone. We verified the effort setting on the wire on smoke runs, reading it back from the sandbox rather than taking it from the request. The toolkit is small. The model has a shell with pdftotext, pandoc, and Python for document extraction, file read, write, and edit tools (the GPT-6 models get a single patch tool in place of write and edit), grep and glob, a todo list, and websearch plus webfetch. Each assignment opens with the same legal role prompt, a senior legal analyst on a multi-step matter, with court-brief word limits stated, and has a three-hour time limit. Every model ran every matter three times.
Scoring
Grading is a signed rubric per assignment. The 122 assignments carry 2,236 criteria, 3 to 54 each, with positive weights for required points and negative weights for penalties such as misstating the record. The assignment reward is earned points over available positive points, clamped to [0, 1]. The matter reward is the points-weighted total across its assignments. The leaderboard is the mean over matters of the mean of three runs.
IRAC classification
A separate three-judge panel assigns each rubric criterion to Issue, Rule, Application, Conclusion, or Other. Each judge sees the matter, the assignment prompt, and the rubric, never a model’s work product, and the majority label is used. The panel agreed unanimously on 78% of criteria and by a two-to-one split on 20%. The 39 criteria with a three-way split were resolved by legal research lead. The classification is fixed per criterion and applied identically to every model.
Judge reliability
A strict majority of three LLM judges (GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.8 Flash) decides every criterion. Each judge reads the full deliverable independently at medium reasoning effort and votes yes or no on that one criterion, blind to which model wrote the answer. Each deliverable is a single markdown file and every criterion is judged in its own call, which removes two known sources of judge drift, the pressure of grading many criteria at once and content a judge cannot see in some file formats.
The judges vote unanimously on roughly four criteria in five, and each seat lands on the majority verdict more than nine times in ten. Gemini 3.8 Flash is the strictest seat for every model graded, including its own output, which is the pattern one would expect if no seat favored its own vendor.
Rescoring the benchmark from any single seat would move no model’s mean by more than about six points. Claude Sonnet 5 grades most leniently for six of the eight models (GPT-5.6 Terra for the other two) and Gemini 3.8 Flash most strictly for all eight. The same three models lead under every seat, though Claude Fable 5.1 and Gemini 4 Argon* swap first and second under GPT-5.6 Terra, and the closely bunched middle reorders.
3. Overall performance: leaderboard, cost, time
Leaderboard
Gemini 4 Argon leads at 50.8%, with Claude Fable 5.1 at 50.6%. The two are the only models above 50%, and Argon wins more matters outright than any other model. Claude Opus 5.5 sits alone in third at 46.5%. Muse Spark 1.3, Grok 4.7, Gemini 3.8 Flash, and GPT-6 Astra form a middle band within a point of one another, around 40%, and GPT-6 Sol trails at 31.9%.
The all-pass column shows how far every model is from being trusted with a matter end to end. No run satisfied every criterion of its matter, and only 9% of individual assignments were completed without a miss. Under an all-pass standard every model would score zero, so the LegalReasonBench leaderboard ranks on the weighted rubric score instead, which gives partial credit for the criteria a deliverable does meet and preserves the signal that separates one model from another.
Reading Gemini 4 Argon’s result
Gemini 4 Argon was released after all other model runs were complete and was added under a near-identical protocol. By then the authors had updated 17 of the 50 matters, so Argon answered a newer version of those tasks than the other seven models did. The head-to-head table is the check on what that means for its result. For each pair of models it takes the difference in score on each matter where both answered the same version of the task, averages those differences, and puts an interval on the average by resampling matters.
On the matching-version matters, Argon is ahead of every other model and Fable is ahead of every model except Argon, with each of those gaps standing clear of noise. Between the two of them the difference is under a point, small enough that it could be an accident of the matters selected. Even with this robustness check, Argon and Fable are inseparable, and the rest of the leaderboard order holds.
Tokenomics
Cost separates models that the score does not. Claude Fable 5.1 spends $19.74 of solver cost per matter, nearly three times Gemini 4 Argon’s $6.71, for a score 0.2 points lower. In the middle band, four models score within a point of one another while their solver spend runs from under a dollar to over $22 per matter. GPT-6 Astra is the most expensive model to run. It writes fewer tokens than the Claude models, but its list price bills cached context at several times the Claude cache rates and doubles every rate once a request passes 272k tokens, and its long web-research sessions carry large contexts. Muse Spark 1.3 is the cheapest at 87 cents per matter. Judging adds $2 to $7 per matter and scales with the length of the deliverables being read.
Time
Gemini 4 Argon works longest, 45 minutes per matter at the median, and GPT-6 Sol shortest at 9. Argon’s end-to-end time is longer still, close to two hours per matter, because its runs spent most of their wall-clock time waiting for the provider rather than working. Wall-clock time depends on provider load on the run dates and is not a property of the model alone.
4. Decomposing model performance
IRAC
Every model spots legal issues better than it identifies governing law or applies it to the facts.
All eight models score higher on Issue than on Rule or Application. The Issue-to-Rule gap runs from 2 to 20 percentage points and the Issue-to-Application gap from 7 to 25. The direction holds for every model, and the Application gap stands clear of noise for six of the eight. The Rule gap is narrower and, for four models, within noise, so the drop after issue spotting is clearest where the model has to apply the law rather than state it. Gemini 4 Argon identifies the legal question about three times in four and then scores in the high fifties on stating and applying the law. The weakest models show the same drop from a lower starting point.
Penalties: the higher-scoring models also fall into more traps
Penalty criteria catch affirmative errors, such as stating an incorrect rule or applying the law in a way the record forbids, and the leading models trip them most. Gemini 4 Argon, Claude Fable 5.1, and Claude Opus 5.5 each trigger about one penalty in four, and GPT-6 Astra about one in eight. The gap is widest on Conclusion and Application, where the three leaders fire penalties at roughly double Astra’s rate.
A closed-universe example shows what the penalties intend to catch.In OTB Gang VICAR Trial Jury Instructions, a closed-universe matter that supplied all the relevant case law and prohibited external research. Models nevertheless applied abuse-of-discretion review standards to the court’s refusal to give the requested justification instruction rather than the applicable de novo standard, triggering a −30 Rule penalty. They supplied a familiar standard associated with jury instructions without identifying the standard governing the particular ruling under review.
This pattern points to superficial reasoning. The leading models include more of what is right without reliably excluding what is wrong. An expansive answer can cover every required argument and still assert a rule that does not apply or a deadline the record rules out, and legal judgment lies in the selection. Listing the possibilities is not the same as determining which one governs. The verbosity analysis later in this section examines how length relates to score and whether filing word limits change the gaps between models.
Research Universe
Legal research failures compound legal reasoning failures
Comparing Rule and Application scores across the two research universes, every model scores higher on closed-universe matters, where the record supplies the governing law and no outside research is required or permitted, than on open-universe matters, where the model has to find the law itself.
Rule scores are 26 to 38 percentage points lower in open-universe matters than in closed ones, for every model, and Application scores 11 to 25 points lower. Claude Fable 5.1 states the governing rule correctly about four times in five when the law is in the record and about two times in five when it has to find the law. Gemini 4 Argon holds up best, with the smallest Rule gap and the highest open-universe Rule score of any model, though its intervals overlap the others.
A trace audit of 64 sampled open-universe failures supports the research reading with a caveat. In most of them the governing authority never entered the trace, but in a fifth the model found the authority, stated it correctly, and misapplied it, and among the matched passes half had cited the authority from memory without retrieving it. Research failure accounts for much of the open-universe gap and not all of it.
The Application gap is smaller than the Rule gap because Application can fail for either of two reasons, a research failure upstream or a synthesis failure downstream. An Application failure in the closed universe comes from synthesis, from reading the rule and the facts and not putting them together, and supplying the record does not fix that, so the open universe adds less to Application than it does to Rule.
Research volume does not close the gap. Every open-universe run made at least one web call, from about 17 lookups per run for Muse Spark 1.3 to over 100 for GPT-6 Astra, and the relationship between lookups and score is weakly positive for four models and negative for the other four, including the two leaders. The models that search the most score no better than the models that search the least.
Verbosity
The highest-scoring models write the most, but length alone does not predict the ranking
Higher-ranked models write longer deliverables
The deliverables here are long, and the longer writers sit at the top of the leaderboard. Gemini 4 Argon writes a median of about 10,600 words per assignment and Claude Fable 5.1 about 9,700, while GPT-6 Astra writes about 5,800 and GPT-6 Sol about 2,700. The ordering by word count and the ordering by score agree at the top and at the bottom. Without controls, longer deliverables score a little higher, and within an assignment the longest third of the answers beats the shortest third by about ten points.
Within the same assignment and model, the length premium is uneven
Comparing deliverables written for the same assignment by the same model removes task size and model identity from the length effect, and what remains is small on average and uneven across models. Pooled, a deliverable twice as long scores about three points higher, nearly all of it from covering more rubric items. For Gemini 4 Argon a doubling buys about 18 points and for Claude Fable 5.1 about 9, and both cover more of the rubric and trip fewer penalties as they write more. For Claude Opus 5.5 and GPT-6 Astra a longer answer scores slightly lower, because the added length brings more penalties than coverage.
Court-facing filings carry a word limit, and models mostly respect it
Twenty-eight of the 122 assignments are court-facing filings, and the role prompt holds them to the word limits of the Southern and Eastern Districts of New York: 8,750 words for a brief supporting or opposing a motion, 3,500 for a reply, and 3,500 and 1,750 on a motion for reconsideration. The 93 memoranda and letters carry no word limit. The harness does not enforce the limits and no rubric criterion scores them.
Every model scores lower on the court-facing filings than on the memoranda. For the three leaders the gap is four to six points, and for the two GPT-6 models it is 11 to 13 points, the only two gaps that stand clear of noise. Every model except Grok 4.7 also writes shorter court-facing filings, with Gemini 4 Argon dropping from a median of about 11,900 words on memoranda to about 8,600 on filings. The two groups differ in task as well as in the limit, so the gap describes the assignments as written. A controlled rerun with the word-limit instruction removed, lengthened the filings by 19% and moved scores by under a point.
Where the limit is stated as a number, on the 16 motion and reply assignments, Muse Spark 1.3, GPT-6 Astra, and GPT-6 Sol never exceed it, four more models comply at least 83% of the time, and Claude Fable 5.1 is the least compliant at 56%, with its median brief right at the limit.
Long-horizon performance
A strong start on a long-horizon matter does not predict a strong finish
In legal practice, early performance on a matter is a reliable signal of later performance. An attorney who frames the issues well at the outset tends to handle the motion and the reply well too, because the same judgment carries through each stage. The multi-step environment and the multi-assignment design of each matter let us test whether models show the same consistency. We decompose each run into its first assignment (A1) and the mean of its later assignments (A2+), and ask whether A1 performance predicts A2+ performance within the same matter. We then ask whether that relationship changes when the record evolves between assignments and when the later assignments depend more heavily on the first.
On average, performance neither collapses nor improves. Gemini 4 Argon and Claude Fable 5.1 tick up slightly after the first assignment, the other six tick down by up to six points, and none of the movements stands clear of noise. The correlation is the more telling number. Within a matter, a model’s score on the first assignment barely predicts its score on the rest, with r between -0.06 and +0.24 for every model and every estimate consistent with zero. The same model can open a matter strongly and finish it poorly, or the reverse, about as often as not.
Explicit dependency between assignments does not strengthen performance continuity
Matters where later work materially requires understanding or building on A1 (34 matters) show no clearer A1-to-A2+ pattern than low-dependency matters (16). The dependency-specific difference in the A1-to-A2+ change ranges from -6.7 to +6.1 percentage points across models, and every interval on that contrast, resampled independently within each stratum, covers zero. The results suggest that models often approach dependent sub-assignments much like separate tasks rather than consistently carrying earlier reasoning forward.
An evolving record does not produce stronger matter-level persistence either
In matters where the factual, documentary, procedural, or legal record changes after A1 (43 matters), average A1-to-A2+ changes remain near zero (Argon* +4.3, Fable +1.3, Opus -1.8, Muse -1.0, Grok -1.2, Gemini -1.0, Astra -2.3, Sol -5.1 points) and the relationship between early and later performance remains weak (r -0.08 to +0.16). Static-record matters show somewhat stronger correlations for several models (r -0.03 to +0.73), but they are only 7 matters (14% of the benchmark) and every static-record estimate is underpowered. Taken together, the results suggest that current models do not yet reliably use the dependencies and accumulating record that define long-horizon legal work.
Later assignments fail on new analysis, not on lost context.
The traces point to the new analysis each assignment demands rather than to lost context. Each assignment starts a fresh session with the earlier work on disk, and models open their earlier deliverable in all but 7 of 1,702 later assignments. In an audit of 48 sampled later-assignment failures, drawn evenly across models, one traced to lost context. Most came from the material the later assignment introduced, and a dozen inherited a gap already present in the first deliverable, so the error carried forward. Each assignment in a long-horizon matter tests a new piece of legal reasoning, and for current models, getting the previous piece right carries no advantage into it.
Tool use and file processing
Models now open every visual exhibit, and still score lower on the criteria that need one
In the preliminary release in May 2026, models often skipped image exhibits and then described their contents anyway, and the rubric penalized the inventions. That gap has closed. Nine of the 50 matters ship image exhibits, and a further 41 PDFs across 22 matters are scans with no text layer, so the model has to look at the page. Every model opened every exhibit on every run.
The score gap remains. Six of eight models score lower on the criteria that depend on a visual exhibit than on the rest of the rubric. Grok 4.7 falls furthest, 27% against 46%, and Gemini 4 Argon scores highest on the visual criteria at 59%, level with its score on everything else. The judges’ reasoning on the failed visual criteria shows a consistent shape. The deliverable describes the photograph or screenshot and does not draw the inference the assignment wanted from it.
The failure comes after access, in synthesizing what the file says
A trace audit of 60 sampled failures on file-dependent criteria, image exhibits and scanned documents alike, drawn evenly across models, traced each failure through three steps: whether the model opened the file, whether the needed content reached it, and whether the deliverable drew the required conclusion. One file was never opened. Six were opened without the needed content reaching the model, for instance a page rendered but never viewed, or handwriting trusted to OCR. In the other 48 the content reached the model and the failure came afterward, most often in the legal inference, where the model saw the exhibit and did not draw the conclusion, and in a smaller number in perception, where it misread the detail. When text extraction came back empty, models rendered the page, ran OCR, or re-read the file. Access is rarely the bottleneck. Synthesis is.
More tool calls do not lead to a better score
Working styles differ widely behind similar scores. Gemini 3.8 Flash fires nearly 150 tool calls per matter and Muse Spark 1.3 makes do with about 50. Half the models lean on pdftotext for document extraction and the other half on Python. The scripted extractors and the search-heavy readers land in the same score band, and within every model the relationship between total tool calls and score is flat or negative.
The same holds for web research. Within open-universe matters the relationship between lookups and score is weakly positive for four models and negative for the other four, including the two leaders. Where an assignment confines the model to the uploaded files, seven of eight models made no web call at all, and Gemini 3.8 Flash did on 6% of those runs.
Conclusion
LegalReasonBench measures legal reasoning on realistic, expert-authored, long-horizon legal workflows. Practicing lawyers built each matter from the case file to the rubric, the facts stay in the files, the law has to be found or read from the record, and the work arrives in two to five assignments that unfold as a matter would. That setting mimics the one in which an attorney’s judgment is exercised, and it proved difficult for every model to complete end to end. The two leaders barely cross 50% of the weighted score, and under the prevailing all-pass standard adopted by other legal benchmarks such as Harvey LAB, every model evaluated here would score zero.
The rubric decomposition and the trace analysis point to a gap between coverage and selection, and that gap is the legal judgment the models currently lack. Every model identifies the issue far more reliably than it states the rule or applies it. The leading models earn the most positive criteria without reliably excluding what is wrong. The models that search the web most score no better than the models that search least. The clearest case is court-facing work, where the stakes are highest and the format forbids the survey. That weakness is particularly consequential in court-facing work, where sound analysis must fit within filing constraints. Every model’s mean score is lower on court-facing filings than on memoranda and letters, yet removing the word-limit instruction produces longer answers without a detectable improvement in scores. The challenge is not simply to fit more analysis into a filing, but to determine which analysis belongs there. Taken together, these findings suggest that the legal judgment demonstrated by frontier models today remains superficial.
These results reinforce our view that the unit for training and evaluating legal judgment should be a realistic, evolving matter. Models should work from expert-curated records and face consequences that increasingly reflect the stakes of legal practice, with each choice tested by the work that follows. LegalReasonBench’s task construction is a step in that direction, providing a foundation for reinforcement-learning environments that train models to exercise meaningful legal judgment.
.avif)