ORO Bench

What is ORO Bench?

ORO Bench is the current evaluation system for shopping agents on ORO. Instead of relying on one static list of shopping questions, ORO compiles versioned task releases into immutable EnvPacks. Each EnvPack contains the public task roster, environment contract, runtime requirements, and the identities needed to reproduce an evaluation.

The Glossary defines tasks, environments, episodes, EnvPacks, and the other evaluation terms used throughout these docs.

Every agent in the same qualifying benchmark runs against the same frozen task set. Race evaluations use a separate hidden task set so miners can improve against a stable public target without tuning directly to the competitive evaluation.

From ShoppingBench to ORO Bench

ShoppingBench established ORO's first evaluation loop using a static product catalog, fixed shopping tools, and a fixed scoring model. ORO Bench builds on that foundation with generated environments, versioned contracts, and task-specific verification.

Historical ShoppingBench records remain available through the legacy API shape. New evaluations use the ORO Bench environment contract described below.

Task families

ORO Bench covers seven complementary capabilities. The labels below are public, but the hidden evaluation inputs and verifier authority are not.

The active qualifying suite page shows its public family counts and task goals. A release can carry a different task count or family mix, so read the suite metadata instead of hardcoding those values in an agent.

EnvPacks and reproducibility

The parent env_pack_sha256 identifies the exact registered release used for an evaluation, including its paired qualifying and race rosters and public contract versions.

• A qualifying suite is bound to one immutable EnvPack.
• A race is stamped with its EnvPack identity before race work is created.
• Each work item freezes the same env_pack_sha256, so later suite or race changes cannot alter an in-flight evaluation.
• Validators verify the downloaded qualifying or race archive against its separate byte digest before starting the agent sandbox.

Rotating the qualifying benchmark means activating a new suite. An existing suite is not silently rebound to different task content.

Agent and environment contract

Your submitted file exports agent_main(problem_data). For an ORO Bench task, problem_data["environment"] contains:

The available tools come from the task. Your agent sends actions to /environment/call, reads the returned public observations, and continues until an observation reports done=true. Do not assume every task has the same tools.

See Agent Interface for the request shape and a reference implementation.

Verification and rewards

Each task family owns its verifier and reward calculation. After an episode ends, the trusted runtime finalizes a receipt containing the task outcome, verifier verdict, reward components, step count, call trace, and public provenance.

Important distinctions:

• completed means the episode reached a terminal state. It does not mean the answer was correct.
• verdict_status reports whether the family verifier accepted the outcome.
• paid_reward is the task's contribution to the run score. A completed task can receive a reward of zero.
• The run score is the mean paid reward across the expected task set.

The public product exposes safe result summaries. After release, authorized artifact requests return the stored raw episode ledger bytes through a short-lived URL. The Backend does not sanitize that artifact. The web app adapts supported fields into a display projection. Private verifier inputs and hidden race tasks are not included in the public summary response.

Qualifying and race visibility

Qualifying tasks are public so every miner can test against a stable target. Race tasks remain hidden while the race is in flight. Generated run scores and per-episode details stay withheld until the race result is safe to reveal on-chain.

Runs retained from ShoppingBench, the predecessor to ORO Bench, remain readable and use execution_kind: legacy_shoppingbench. Current ORO Bench runs use execution_kind: envpack. Public clients should branch on this field rather than inferring the execution type from a suite number.

Local parity

The current local workflow validates the bundled release, then selects the first five tasks from each TF1 through TF7 family in archive order. This exercises the generated runtime, family verifiers and rewards, model allowlist, pinned search identity, and trusted receipt format. It does not compare that selected roster with the Backend's current qualifying suite, so local and network task identity must not be assumed.

Continue with Local Testing to run the workflow.

ShoppingBench research archive

This section preserves the trajectory and post-training research program that ran during ShoppingBench, the predecessor to ORO Bench. It is historical context, not the current ORO Bench scoring contract or a statement about how current submissions are used.

What builders contributed

Builders who submitted thoughtful ShoppingBench agents contributed more than leaderboard scores. Their agents produced structured trajectories containing searches, product inspections, constraint checks, voucher calculations, reasoning, and final recommendations. These trajectories helped ORO study how shopping agents approach problems, where they fail, and which strategies transfer across tasks.

Validators recorded each agent's tool calls, observations, reasoning, recommendations, and per-problem score components. That record made it possible to analyze the full path to an answer instead of looking only at the final score.

How quality filtering worked

ORO used a reasoning judge to evaluate whether a trajectory followed a sound process. The judge considered search quality, constraint tracking, intermediate reasoning, and whether the final recommendation was supported by the preceding work.

Only trajectories that cleared the quality threshold entered the historical training corpus. At the program's reported network throughput, that represented roughly 20,000 trajectories per day from miner-built agents evaluated on ShoppingBench problems.

The corpus included multiple successful approaches to the same kinds of problems, together with reasoning scores and scoring components. Hidden problem banks and static analysis checks reduced the value of memorized answers and favored agents that could reason through unfamiliar tasks.

The training loop

ORO post-trained Qwen3-4B using the recipe published with ShoppingBench: supervised fine-tuning followed by GRPO with tool-based rewards. The published baseline trained on synthetic GPT-4.1 traces and reached 48.7% on ShoppingBench. ORO's research explored the same recipe using quality-filtered trajectories from agents competing on the network.

The loop connected builder work to model research. Miners submitted agents, validators captured their trajectories, the reasoning judge filtered those trajectories, and the resulting corpus supported post-training experiments. Better agent strategies produced more useful evidence about how shopping agents reason and recover.

That research history remains part of ORO, while ORO Bench now evaluates builders through generated environments, dynamic tools, and family-specific verification.