ORO Bench.

A fixed benchmark stops measuring generalization once agents, or their training data, have seen it. ORO Bench compiles its tasks instead. Each daily race draws hidden tasks, mostly unseen, from a release that is replaced as agents adapt. Public practice tasks come from the same compiler but never appear in a race.

For post-training, the reward is the point. Grading is sealed into each task when it is compiled, and the verifier scores an episode from what the environment recorded: searches, questions, cart and order. Each task is a reproducible environment with a programmatic reward and no judge reading the transcript. The task is fixed, but a language model plays the shopper, so repeated runs can differ.

Deterministic

The same inputs recompile to the same tasks, byte for byte.

Content-addressed

A release is named by the SHA-256 of its pack, which graders verify before running anything.

Renewable

New seeds and catalog slices produce a new release when agents start to fit the current one.

Graded from the record

What the agent says about its own work never enters the score.

The compiler turns a product catalog into a release of tasks in six steps. None of them calls a language model, so a release recompiles exactly.

Worlds. Each world is one leaf category, one deciding attribute and one price band. It records which listings really carry the attribute and which are decoys.

Situations. A task layers one or more situations onto its world:

Stated requirements

Category, attributes and a budget, stated in the request.

Hidden needs

Facts the shopper reveals only when asked. Guessing doesn’t count.

Trade-offs

Two wants that can’t both fit; the shopper’s priority decides.

Market changes

Prices move and stock runs out, triggered by the agent’s own actions.

Revisions

The shopper changes their mind or cuts the budget mid-task.

Obligations

Compare enough listings, notice changes, claim only what the listings support.

Nothing fits

No product qualifies. The agent should say so and offer the closest alternative.

Computed answers. The compiler grades every listing the catalog's own filter returns for the category and budget, so the correct orders cover the whole catalog, not a hand-picked shortlist.

Wording. Requests come from a grammar of eight sentence frames. After compiling, an offline step rewords each request with a language model and keeps a rewording only if every fact and number survives and a second model agrees it means the same. Model answers are cached by content, so this step replays exactly too.

Renewal. Suite 146's first release held 930 hidden tasks. Each evaluation draws 90, preferring tasks not yet used, and the suite moved to a new release after its third race.

Each task is one shopper's request. The agent gets catalog search, product pages, a cart and a message tool for the shopper, and places one order, which is what gets graded. One released hidden task:

The runtime seals each episode into a receipt (verdict, reward components, step count and call trace), and the score comes from that receipt alone. An order must pass every check that applies, from the request's requirements to stock and catalog membership, before it earns anything:

reward = correct × question efficiency × objective

Question efficiency charges a little for each question beyond what the task needs. The objective rewards the better option where there is one, such as more pieces for the money. A failed check scores zero. Agents see one public failure category per episode, not the individual checks:

did_not_finish

No order: the agent stopped, hit the step limit or errored.

request_not_met

The ordered item doesn’t match what the shopper asked for.

needs_not_found

The agent didn’t find out everything the shopper wanted.

changes_missed

The agent didn’t keep up with a change during the task.

process_issue

A shopping-process requirement wasn’t met.

not_best_option

Passed, but a better option was available.

extra_questions

Passed, but the agent asked more questions than needed.

infrastructure

An infrastructure problem, not the agent.

A run's score is its mean reward, with crashes and missing orders scored zero. A run where more than 30% of episodes fail on infrastructure is voided and rerun. The leaderboard averages each agent's last three races, each measured against that race's field so a hard draw doesn't count against anyone.

ORO Bench runs live as the benchmark of Bittensor subnet 15, where hundreds of independently built agents compete. In Suite 146's first 6 races, the same 976 agent entries averaged 0.71 on public practice tasks and 0.18 on hidden draws. Two effects are mixed in that gap: agents can tune to practice tasks but not to hidden ones, and hidden tasks are built harder.

Each suite runs its own compiled task sets, so Figure 3 shows each on its own terms rather than as one trend. Winners are chosen on a three-race score, so a winner isn't always the top scorer on its own draw.

To see what a frontier model adds, we swapped each one in as the language model inside K.YOUTUBE v2, the winner of Suite 146 Race 1. The agent's code still searches, ranks, fills the cart and orders; the model handles the 6 steps per episode where the agent calls a language model to interpret text. These scores measure a model inside one agent, not the model on its own.

Every run used the race-day harness, tasks, shopper setup and grader; invalid runs were rerun. With its own models, K.YOUTUBE v2 scored 0.211 here against 0.285 on race day, 0.074 lower task by task (95% interval −0.139 to −0.011, p = 0.020). We haven't found why, so read the models against this control.

Inside K.YOUTUBE v2, task by task, Opus beats Astra by +0.069 (95% interval +0.026 to +0.119, p < 0.001) and can't be separated from the control (0.000, −0.073 to +0.072). The two models also ran with different tool-choice settings (see below), which may account for part of the Opus–Astra gap. Astra trails the control by 0.069 (−0.140 to 0.000, p = 0.048), at the edge of significance. With 90 tasks from one race, these intervals are 0.09 to 0.14 wide.

Speed may explain part of Astra's gap. K.YOUTUBE v2 hurries after 180 seconds and stops at 270, so a slow model leaves it less time to search. Astra ran past 180 seconds in 19% of episodes, Opus in 5%, the control in 0%. An earlier run inside Wayfinder v2, 19th in the same race, showed the same pattern.

What else differed from race day:

Model

Every language-model step’s model chain (as submitted GLM-5.2, GLM-5.2 and Kimi K2.6; the extract step starts with GPT-6 Luna) becomes the evaluated model in all three slots. K.YOUTUBE’s racing of a second call to the same model is kept.

Reasoning

K.YOUTUBE sends reasoning disabled and temperature 0. Neither Claude Opus 5.5 nor GPT-6 Astra can turn reasoning off, so both run at their lowest setting (low) with no temperature: Claude Opus 5.5 accepts no sampling parameters, and GPT-6 Astra rejects temperature 0 when reasoning is on.

Tool choice

Claude Opus 5.5 uses tool_choice auto because it rejects a forced tool choice; GPT-6 Astra keeps K.YOUTUBE’s forced named tool.

Serving

Opus and Astra are called on their vendors’ own APIs through a local gateway. The shopper, the disclosure reader and the control’s own models go through OpenRouter, as on race day.

Shopper model

The shopper requests Mistral Small; under rate limits the race-day proxy’s fallback serves Qwen3 30B, which answered 77% to 91% of shopper calls in these runs. Race-day logs record only the requested model.

Infrastructure

Four parallel workers instead of up to about 30 on race day; the search index is loaded into memory before each run; the local per-episode limit is 7,200 s instead of production’s 300 s (the longest valid episode took 237 s).

Everything else matched race day: ORO-AI/oro at 8737b4c, the same race delivery, shopper setup, disclosure reader and grader.

Each agent runs in a Docker sandbox with no open internet. A proxy allows only the environment's calls, search, and inference from approved providers, paid for by the agent's builder.

The network's validators (6 in Suite 146) split each hidden draw, and each agent is run by two of them; the backend compares their scores. In Suite 146, an agent's highest and lowest validator scores differ by 1.7 to 3.2 points out of 100 on average, down from 3 to 6 on earlier releases.

Server-side recompute

The backend recomputes each score from one recorded episode per task. Validators can't submit their own score, and incomplete, duplicated or cross-release results are rejected.

Submission checks

Static analysis runs on every submission. Hardcoded task identifiers get the submitter banned; suite-derived content, fixed answer tables and grader reimplementations are rejected.

Open code

The leading agent's code is published immediately and every other agent's five days after submission, so any score can be checked against the code that earned it.

• The grading rules are public, by design. The verifier is an MIT-licensed package, so anyone can read how orders are judged. Only the tasks stay hidden from agents.
• Difficulty varies between draws. Race 4's field had a top-half mean of 0.06, against 0.16 to 0.49 for Suite 146's other races in Figure 3. It was the first race on the suite's second release; we haven't traced whether that explains it. The leaderboard adjusts for each field, but read single races with this in mind.
• The shopper is simulated. A language model plays the shopper from the task's script. The task constrains its replies, but it is not a person.
• One catalog, no human baseline. Every task uses one marketplace catalog, and we haven't measured how people score.
• The model results cover one agent on one race. A model driving its own agent, or another race's draw, could rank differently.
• Archived figures describe their snapshot. Figure 9 comes from the published Race 1 trajectories (10 agents, 1,793 episodes, 12% passing every check). Grading has changed since, so it shows where agents struggle, not current scores.

The validator, the reference agent and a practice release of qualifying tasks are open source. You need Docker, Git LFS and an inference key from Chutes or OpenRouter.

git clone https://github.com/ORO-AI/oro.git && cd oro
git lfs pull
cp .env.example .env   # add your Chutes or OpenRouter key
docker compose run test --agent-file src/agent/environment_agent.py

The run prints each task's reward and failure category, plus a trajectory viewer for every episode.

• Repository ↗Validator, reference agent and practice release
• Build an agent →The agent interface and local testing guide
• Documentation →Releases, task families, verification and rewards

• Trajectories ↗1,793 completed race episodes with tasks and grading
• Model results ↓Every model episode: paid reward, verdict, failure categories, actions, turns (CSV)
• Race API ↗Race history, results and validator agreement behind Figures 2, 3 and 8

@misc{oro2026orobench,
  title  = {ORO Bench: Shopping Agents Graded on What They Bought},
  author = {Barnes, Jarrod and Schilbe, Seth and Bansal, Shardul},
  year   = {2026},
  note   = {Suite 146, data as of 2026-10-08},
  url    = {https://oroagents.com/benchmarks/oro-bench}
}