07. Durable jobs

Faster attempts still left people responsible for tracking evaluations, so the service now manages that work against a coverage target.

One request specifies tasks, models, clean results per task-model pair, and a budget. Each pair forms a cell in the coverage matrix, starting with one rollout.

From one request, the service manages the work needed to reach the coverage target and reports anything it could not measure.

The controller recalculates coverage from the database on each pass, avoiding separate counters that can drift. A clean result must succeed, receive a score, and have no failure class.

The controller chooses how to recover based on the error: rate limits and lost sandboxes slow retries without adding strikes, while agent deaths add strikes. Repeated strikes quarantine the cell, and broken tasks cause immediate quarantine.

The service excludes quarantined cells from results and reports each one with its reason. Unknown errors add strikes so that quarantine exposes the problem before repeated retries spend more money on it.

The controller finishes the job by reaching the coverage target for each cell or quarantining it with a reason.

08. Silent failures

A clean result depends on the service knowing when a run has failed, and the hardest failures to see are the ones that end with a score.

An agent can work in its sandbox for hours. If its connection drops, a time limit expires, or the sandbox kills its process, the trial can end without raising an error. No answer file exists, but the verifier still runs, the judge scores the missing answer zero, and the worker reports a successful run.

That record looks exactly like a wrong answer, so the benchmark counts a dead agent against the model. In one maximum-effort campaign, stalled connections left 75 of 89 zero scores on one benchmark with nothing on the record to explain them. Once the cause was fixed, the same model's maximum-effort score on another benchmark rose from 0.18 to 0.43, level with its default setting.

The worker now checks how the agent ended before it accepts a grade. A nonzero exit, a harness session that died, a stall, a timeout without progress, or a lost answer file makes the attempt a failure with its reason and no score, and the controller retries it. A run that earned any credit keeps its score.

A dead agent and a wrong answer used to leave the same record.

A September audit of 2,085 benchmark runs across twelve model and benchmark pairs found no false zeros left in the scores.

Other agents stall without dying. Our earlier guards watched for activity, such as file changes, output, and bytes on the wire. An agent stuck retrying a failing request refreshed all of them about every 40 seconds, so it kept running until its two-hour limit and then scored zero.

A watchdog inside the sandbox now measures progress instead: a completed model response or a step the harness saves. An open stream, a running tool, or CPU use shows that the agent is alive but never counts as progress. The watchdog stops an agent that keeps failing for 15 minutes without completing a step, or one that shows neither progress nor a sign of life for 40 minutes. It saves the transcript and records the run as stalled, and the controller retries it up to twice.

The watchdog stops an agent that has stopped making progress, even while it still looks busy.

In its canary, all 14 maximum-effort runs finished without a false stop, and the longest healthy gap between progress signals was six minutes. One task that had never finished at that effort completed for the first time.

The watchdog also streams the agent's transcript, task list, and vitals out of the sandbox while the run is in progress, so a stall can be diagnosed before the run ends.

09. Benchmark campaigns

Benchmark campaigns freeze the task set, task versions, and judge before execution. The service publishes scores with bootstrap intervals only after every cell has a measurement or an explanation.

The results ledger retains earlier entries, with corrections in new rows that supersede the originals.

For a new model, one command starts a smoke test and then the full benchmark matrix. An agent checks for infrastructure errors recorded as zero scores and unintended task-version changes, then drafts the results for review.

The ledger stays locked until the matrix is complete.

10. Continuous optimization

We have reduced much of the overhead in admission, worker startup, and sandbox setup. As workloads grow, we continue to optimize that path and coordinate each job with the capacity available to it.

When we launched a large batch of maximum-effort runs of a newly released frontier model, our service admitted and started every run within minutes. Requests above the model's rate limit had to wait, showing why fast starts also need pacing.

The controller caps each job and slows requests as rate-limit refusals increase. Capacity planning is now an early part of our release-day discussions with labs.

Faster starts need pacing so each job makes use of available capacity without building up waiting requests.

Each environment bills to its own provider account. Agents use short-lived, scoped tokens, while an external broker sends requests under the environment's provider identity. Provider keys never enter the sandbox, and no shared fallback key obscures attribution.

11. Release-day harness

Release day also tests the agent harness. We run every model through the same open-source harness, OpenCode, so benchmark scores compare models rather than agent frameworks. Each harness build ships with a catalog of the models it knows, and a model released this morning is not in it.

For a model it does not know, the harness falls back to defaults. It capped each response at 32,000 output tokens, so a model reasoning at maximum effort could reach the cap mid-thought and end its session without writing an answer. It could also drop a reasoning setting it did not recognize, so a run requested at maximum effort ran at a lower one. Neither raised an error, and both looked like a weaker model.

Rerunning the 38 of 150 runs that one frontier model had lost to the cap raised its benchmark score from 0.33 to 0.48.

Our first fixes were per model: proxies that rewrote the cap and lists of models to lift, which took 30 changes between July and October and still drifted. An adapter in the service now applies one rule to every run. Each run receives the model's documented output maximum, or 128,000 tokens when the provider has not published one, and the sandbox refuses to start the harness without it. Reasoning settings use the key each provider's SDK forwards, and a new model's limits can travel with the launch request, frozen into each run, without waiting for a deploy.

An adapter gives the harness a new model's real limits before the first run.

Before the full launch, a two-task smoke run reads the request back from the wire: the effort that reached the provider, the declared output limit, the presence of reasoning, and the absence of a length stop. The last three models we benchmarked at release had results within about a day, and each still needed a same-day code change for its limits. The next one will not.

12. Enhanced logging

We record stage timings for each attempt to distinguish service delays from agent execution and provider limits, rejecting older evidence. Unknown durations remain null so that missing data cannot lower a reported median.

Missing cost receipts likewise leave costs unknown rather than zero.

Our telemetry platform shows live rollout stages and groups durations by environment and errors by cause, giving a model the evidence to draft the initial triage.

13. Accelerating iteration

Next, agents will come installed in task images, and prebuilt images pinned by digest will serve multiple sandbox providers. Admission will respond directly to provider pressure, so the controller can determine how much work each model can accept.

We’ll share more soon as we continue to optimize, with a larger improvement already on the roadmap.

References

Harbor · the task format and trial runner our environments build on.

Modal Sandboxes · sandbox and function documentation.

Daytona · sandbox documentation.

OpenCode · the open-source agent harness our benchmark runs use.