Most evaluation of financial AI asks whether the model got the answer right. Can it compute the ratio, read the earnings report, interpret the filing. Those are the right questions in the wrong order, because before a system can reason about anything it has to decide which document it is reasoning from. That step is rarely measured on its own, and it is where a surprising share of the failures live.

This matters more in finance than almost anywhere else, for a reason that sounds mundane. The corpus is full of near twins. Last year's annual report and this year's share a company name, a product list, a vocabulary, and a structure. To a search system they look almost identical. To an analyst they are not interchangeable at all.

Relevant is not authoritative

The distinction worth drawing is between a document that is relevant to a question and a document that is authoritative for answering it. Ask for a pharmaceutical company's revenue breakdown by drug for fiscal 2024 and the prior year's annual report will look like an excellent match on every surface signal. Same company, same drugs, same terminology. It is a legitimate document. It is simply the wrong period. Nothing in the title or the URL tells you that, which is the point: even an expert cannot resolve it by looking at the metadata alone.

So who decides what counts as authoritative? Not an engineer, and not anyone outside the domain. At micro1 that judgment comes from practitioners who work in financial institutions, writing realistic diligence questions and identifying which documents actually answer them. Authoritative does not mean the document that gets at the gist. It means the source you could defend in a regulated setting, and sometimes there is more than one.

I have a long-standing interest in this problem from a different direction. When I began training models at micro1 in 2024, I was working in policy, where several sources can be credible at once and some of the disagreement between them is ideological rather than factual. Finance looks like the easy case by comparison, since so much of it is filed, dated, and audited. It is not as easy as it looks. The hierarchy is real but partial, and the unreported metrics, the ones that come from interviews and trade press rather than filings, put the analyst right back in the position of adjudicating credibility.

The ranking gap

The most interesting result from the Cortex retrieval benchmark is a gap between two questions that are usually treated as one.

Ask whether an authoritative document appears anywhere in the top ten results and the systems look excellent: between 92 and 99 percent across the architectures tested. Ask the stricter question, whether an authoritative document is ranked first, and performance falls to somewhere between 42 and 69 percent depending on the architecture. The system finds the right document. It does not reliably know that it is the right one.

That gap held across keyword search, dense retrieval using embeddings, and hybrids of the two. More sophisticated architectures narrowed it. None of them closed it, which suggests the problem is not a weak similarity function but a missing concept. Semantic similarity has no opinion about fiscal periods.

The engineering implication is useful precisely because it is unglamorous. If the authoritative document is already in the top ten, retrieving more documents will not help. What helps is metadata-aware retrieval and authority-aware re-ranking: carrying reporting-period and document-type information into the chunks themselves, so that a passage which is semantically relevant can still be distinguished as 2023 rather than 2024. The fix is at the ranking layer, not the recall layer.

This also matters because of how downstream systems consume evidence. Some pass the top three results to the model, some the top ten, some re-rank first. High coverage inside a window the model never fully reads is not the same as putting the right source in front of it.

Where the cost actually lands

The obvious cost of bad retrieval is a wrong number in front of a client. The real cost is slower than that, and larger.

Financial work is derivative of its inputs. An investment decision or a piece of client advice is a judgment built on sources, and the sources are themselves the object of scrutiny. Take EBITDA for a given company: there may be ten places to find the figure, and an analyst will use the 10-K rather than an aggregator. If the number from the weaker source happens to be identical, citing it still puts the entire analysis in question. Provenance is not a formality in this industry. It is part of the claim.

The deeper cost is to trust, and trust behaves asymmetrically. A user who is handed a confidently wrong retrieval does not merely discard that output. They stop bringing the hard use cases to the tool, which means the capability stops compounding for them at exactly the point where it was becoming useful. The models are powerful and the models are useless if nobody routes real work through them.

Which is why the product layer carries as much weight as the model layer. Citations and auditability are not decoration; they are the mechanism by which a domain expert stays in the loop cheaply, checking lineage rather than redoing the search. And the stakes rise as these systems move from answering questions to completing work. A single question has one place to go wrong. A multi-step task has many, and the trust required compounds across every one of them.

What a benchmark is for

Buyers at financial institutions ask about two things before they sign: accuracy and latency. The first is hard to answer honestly without a shared measuring stick, which is most of the argument for building one.

A retrieval benchmark does three jobs. It lets firms compare vendors on the ability to surface authoritative evidence rather than merely similar text. It gives engineering teams a regression test, so that swapping an embedding model, changing a chunking strategy, or adding a re-ranker produces a measured answer instead of a hopeful one. And it supports deployment decisions by separating two failures that look identical from the outside: the system that reasoned badly, and the system that reasoned well from the wrong document. Those have different fixes, and you cannot tell them apart by reading the output.

The honest limitation is that we have measured the ranking and not yet its consequences. We know how often the authoritative source fails to come first. We have not yet quantified how much that costs in final answer quality once it flows through a production system. That is the next piece of work, along with agentic approaches that separate identifying the right document type from selecting the right passage within it. The direction of travel is clear enough: retrieval is becoming less a single similarity score and more a sequence of evidence-selection decisions.

The junior analyst question

My own worry about all of this is not that senior judgment disappears. It is the entry-level ladder. If retrieval and drafting are the tasks that taught junior analysts what good looks like, automating them efficiently could remove the rungs people climb.

The more convincing answer I have heard is that judgment and taste become more valuable as the mechanical work gets cheaper, and that the analyst's job becomes more interesting rather than smaller: owning a fuller cycle of work, generating revenue earlier, still answerable for what goes up the chain. That depends on a condition we should state plainly. Someone has to still learn why the 10-K beats the aggregator, and that understanding has historically been acquired by doing the tedious version first, under mentorship, inside a firm. If the tedious version goes away, the learning has to be designed rather than absorbed.

That is a reason to reframe what we expect from an entry-level role, not a reason to protect the old one. The legacy logic of the ladder was never the point. What it produced was.

Which brings this back to where it started. The question of which document deserves to be trusted is the same question whether a human or a system is asking it, and it is answered with domain judgment rather than with similarity. That is the argument for measuring the retrieval step directly, and for keeping the people who know the difference close to the systems that have to learn it.

This piece draws on a conversation hosted by the micro1 Forum, featuring Shaudi Mahdavi, Member of Technical Staff at micro1, and Strib Walker, Head of Product at Rogo. The full benchmark report is available here.