Back
RealWorkDecision
Evaluating AI models' ability to understand intent and make decisions inside a real company.
%20(1).webp)
Introduction
Current AI benchmarks often test models in contrived and in highly synthetic environments on their ability to carry out straightforward tasks. Models have saturated these benchmarks very quickly, but still often lack capability in real-world deployment, where there is far greater complexity and task horizon.
We introduce RealWorkDecision, a benchmark that assesses how capable models are at making decisions when placed to act in an actual company. micro1 Company Data Partnerships (CDP) enables us to build these worlds from real company data rather than synthetic environments. The benchmark places agents in a frozen world, assembled to match what a real employee saw across all applications they have access to, exactly one second before communicating a business decision to a colleague. We then evaluate how accurately frontier models are able to navigate their entire context and predict their decision.
Corpus overview
The corpus holds 1,222,081 records from 63 employees across 16 applications, spanning February 2015 to October 2026, and 53 worlds were frozen from it.
Methodology and World Assembly
Using our access to real world data via our company data partnerships, we assembled numerous employee worlds. Each world was created by taking the large company data export and carefully mapping employee relationships across application and time. We then mine the data using our proprietary flow map for moments where key decisions were made and freeze the decision-maker’s world at that instant, carefully auditing to ensure data matches exactly to the permissions the employee had at that time and no data is leaked from future application states.
For each decision made in the corpus, a hand-written rubric that evaluated the model’s prediction was created by a domain expert and carefully audited using flow qc, ensuring no valid alternative solutions are penalized. Additionally, the evidence for every single decision made was carefully traced through the corpus, and any decision which lacked sufficient context was filtered out. Each task created had substantial background information from the corpus to help inform the decision.
Example employee worlds
Results
Cross-reasoning testing
Example tasks
Failure modes
Conclusion
We believe that in order to make AI models deliver value for the real world, they must be trained and evaluated in environments that are much more real-world than they are today. Through our company data partnerships program and our state of the art privacy transformation model, we are scaling realism and horizon in order to make AI models better able to act in the real world.
.avif)