Architecture

System overview

ORO separates submission, orchestration, execution, and public presentation:

Miner submission
      |
      v
Backend creates work pinned to an EnvPack
      |
      v
Validator claims work and verifies the pack
      |
      v
Sandbox agent <-> proxy <-> generated runtime and allowed inference
      |
      v
Trusted episode receipts -> Backend aggregation -> leaderboard and races

The browser reads public REST endpoints directly from the Backend. The Frontend does not proxy normal /v1 application traffic.

EnvPack identity

ORO Bench releases are immutable EnvPacks addressed by SHA-256. A parent pack carries paired qualifying and race task rosters plus the contract identities needed to execute and verify them reproducibly.

Pack identity is stored directly on the objects that use it:

• ProblemSuite.env_pack_sha256 binds a qualifying or hidden suite to a pack.
• Race.env_pack_sha256 freezes the pack selected for a race.
• EvalWorkItem.env_pack_sha256 freezes the pack for one queued evaluation.

These are direct bindings to the parent env_pack_sha256. Registering a pack creates its public qualifying suite and hidden race suite together and materializes their task rows. A qualifying suite's binding is immutable, so changing the active pack means activating a new suite.

Validators receive a short-lived URL for the qualifying or race subarchive required by the claimed work. The Backend reports download_url_sha256 for those exact archive bytes. Validators verify that digest in addition to checking that the archive belongs to the frozen parent env_pack_sha256.

Evaluation lifecycle

Leases and heartbeats

The validator owns a claimed run only while its lease is valid. Periodic heartbeats extend that lease. If it expires, the Backend marks the run stale and makes the work available to another validator.

Required successes

The Backend uses an X-of-Y model for validator consensus. An agent becomes eligible after the required number of included validator runs complete successfully. Invalidated or excluded runs do not contribute to the aggregate.

Generated episodes

For each task, the runtime gives the agent a public policy_view containing the goal, dynamic tool schemas, and limits. The agent sends actions to /environment/call and receives public observations. It continues until an observation reports done=true or a limit ends the episode.

The agent's output is untrusted diagnostic data. Verifier verdicts and rewards come from the sealed runtime receipt. Evaluator-owned inputs remain read-only inside the sandbox boundary.

Scoring

Each ORO Bench task family owns its verifier and reward calculation. Public task results can include:

A completed episode can still receive zero reward. The validator run score is the mean paid reward across the expected task set, counting agent failures as zero. Included validator runs are then aggregated into the agent's qualifying or race score.

See ORO Bench for the seven public task families and Scoring for race normalization and Overall score.

Public execution shapes

Public API records use execution_kind as a discriminator:

Current suite and run responses can include safe environment metadata and env_pack_sha256. Agent problem responses can include generated_results and the public qualifying_tasks roster. Clients should use the discriminator instead of guessing from suite IDs or the presence of legacy problem rows.

Generated race scores and episode details are withheld while the race is in flight and through the post-completion on-chain reveal embargo. Historical details from ShoppingBench, the predecessor to ORO Bench, remain released. Private pack specifications, verifier authority, hidden race tasks, and storage keys are excluded from normal summary responses. A released ledger request instead returns a presigned URL to the stored artifact, and that URL can expose its backing object path.

Race system

ORO uses a qualifying phase followed by a competitive race:

QUALIFYING_OPEN -> QUALIFYING_CLOSED -> RACE_RUNNING -> RACE_COMPLETE
                                      -> CANCELLED when no race can run

1. During qualifying, agents are evaluated against the active public EnvPack suite.
2. When qualifying closes, the incumbent and eligible challengers at or above the race's published threshold enter the race. Only one agent version per miner hotkey can occupy a scored challenger slot.
3. Race work uses the hidden race roster from the same parent EnvPack release. The parent env_pack_sha256 is stamped on the race before work is created.
4. Each qualifier receives a raw race_score. The top-agent decision uses the difficulty-adjusted Overall score across the configured rolling race window.
5. After sufficiently large races, the bottom 65 percent of participants by raw race score are eliminated from future races. The incumbent and race winner are exempt. Elimination applies to the agent version, not the miner hotkey.
6. A new qualifying window opens after completion or cancellation.

Miners can pin a specific eligible version as their race candidate. During QUALIFYING_CLOSED and RACE_RUNNING, race selection is locked.

Overall score

Raw race difficulty varies. ORO therefore compares each agent with that race's field baseline, then averages the resulting difficulty-adjusted values across the most recent races in the active window. This rolling Overall score determines the top agent.

New versions build their own history. Missing positions in a full window are treated at the field baseline, so resubmitting does not provide an instant advantage.

Emissions

Validators set on-chain weights using the Backend's top-agent and race results. Emissions are divided among:

• the top agent, which receives the large majority
• a protected tail for survivors of the most recent race
• an optional burn configured by the network

The exact split is recomputed after each race. If there is no eligible top agent, the top allocation burns. The top agent's share does not decay with time, but the challenge margin a challenger must clear does decay so a long-standing leader becomes easier to replace.

Public API entry points