View all research work

Back

November 16, 2026

Every task, checked against every public benchmark

Jacky Guan

Jacky Guan

,

Member of Technical Staff, micro1

Rumi Elias Calles

,

Director of Research Eng, micro1

Illustrative. Each square is one task or benchmark item. A new task is compared with public benchmarks across modalities and with existing flow tasks.

Why it matters

A benchmark score only means something if the model has never seen the test.

When a benchmark item ends up in training data, a model can score well by remembering the answer, and the score alone does not show it. Duplicates inside a dataset do quieter damage: the same task counted twice makes a dataset look larger than it is and gives one problem extra weight.

Contamination is usually measured after training, on the finished model. flow qc checks the data itself, when a task is created, so a leaked or repeated item never reaches training.

How it works

Every task goes through the same four steps.

  1. Capture

    Text, code, files, images, audio and video are read when the task is submitted.

  2. Index

    Each task becomes a fingerprint that covers every modality it contains.

  3. Compare

    The fingerprint is checked against all public benchmarks and Flow tasks in seconds. Flow QC reviews matches.

  4. Flag

    A confirmed match is held back with the matching item attached. The rest pass.

The engine is built in house. It fingerprints every modality a task can carry, so a pasted prompt, a reworded problem, a re-plotted chart or a cropped screenshot still leads back to its source. The internal flow qc agent then reads both items in a close pair and decides whether the new task is a copy, a rewrite or a coincidence.

Always current

New public benchmarks appear every week. micro1 collects them as they are published and adds them to the index, so a task is checked against a benchmark released yesterday as well as the long-standing ones.

Across a batch

Matches leave the batch. Everything else keeps moving.

Illustrative. Counts show the mechanism, not a real batch.

What good data is

Good training data teaches a model something it could not already do. A task copied from a benchmark teaches nothing new, and it makes the model look better at the one place progress gets measured. A task repeated twice teaches the same lesson twice and takes the place of one that was never written.

That makes the value of a task relative. The same problem can be worth a great deal on its own and nothing once it sits in a public test set. A task can only be judged against everything a model may already have seen.

Benchmarks are how the field measures progress, and they only work while models have not seen them. Checking every task against every public benchmark keeps micro1's data out of them, so a model trained on it earns its scores.