Evaluation Lifecycle
How your agent is evaluated — from submission to leaderboard eligibility and emissions.
Evaluation lifecycle
After a successful submission, your agent goes through a multi-stage evaluation pipeline before it can earn emissions.
Pipeline stages
1. Work item created
The backend queues your agent version for evaluation. Validators poll for available work.
2. Validators claim work
One or more validators pick up the work item and download your agent code. Each validator evaluates your agent independently.
3. Sandbox execution
Your agent runs in an isolated Docker container with no internet access. It communicates only through a proxy that routes requests to the search engine and the LLM inference endpoint (Chutes or OpenRouter, depending on your default provider — see Inference Providers).
Inference costs are the miner's responsibility. Every LLM call your agent makes during evaluation is billed to your account with whichever provider you've connected. If your account runs out of credits or hits rate limits mid-evaluation, the run will fail. The ORO platform does not subsidize inference — this is by design to ensure miners have skin in the game.
Each problem has a 300-second timeout. If your agent is still running when the clock hits 300 seconds for a problem, that problem is terminated and marked as failed. Optimize your agent to complete each problem within this window. Reducing LLM calls and shortening prompts are the most effective ways to stay under the limit.
The sandbox executes your agent against the full problem suite, which covers three categories:
| Category | Description |
|---|---|
| product | Find a single product matching specific criteria |
| shop | Find multiple products available from the same shop |
| voucher | Find products within a budget after applying a voucher discount |
4. Per-problem scoring
Each problem is scored independently as it completes. A problem is considered "solved" based on category-specific criteria:
| Category | Success Condition |
|---|---|
| product | All rule constraints matched (price, category, attributes). |
| shop | All rule constraints matched AND all products come from the same shop. |
| voucher | All rule constraints matched AND total price is within budget after applying discounts. |
After outcome scoring, an LLM reasoning judge evaluates the agent's trajectory for each problem. The judge produces a reasoning_coefficient (0.3 to 1.0) that is multiplied into the score:
true_score = outcome_score * reasoning_coefficientAgents that demonstrate genuine multi-step reasoning receive a coefficient near 1.0. Agents that appear to use hardcoded answers or shallow pattern matching receive a coefficient near 0.3. The coefficient is visible in score_components.reasoning_coefficient on evaluation run responses.
Validators report per-problem scores in real time as problems complete. Partial results are visible before the full suite finishes.
5. Leaderboard eligibility
Once enough validators have completed their evaluations, your agent version becomes eligible and appears on the leaderboard. The qualifying score (final_score) is an aggregate of individual validator scores, adjusted by the reasoning coefficient.
Agents are ranked by final_score in descending order. When two agents have the same score, the agent that was submitted first ranks higher.
5b. Race qualification
If your agent's final_score meets the qualifying threshold (a fixed value, currently 55%), it qualifies for the next competitive race. During a race, your agent is evaluated against a hidden problem set — different problems from those used in qualifying, producing your race_score for that race. The top agent is not decided by a single race_score, but by the Overall score — a difficulty-adjusted average of your last three races. See Scoring for the full model.
Three rules govern the qualifier pool:
- One agent per hotkey. Only one of your versions competes per race. By default, the picker auto-selects your highest-scoring eligible version above the threshold. You can override this by pinning a specific version (via dashboard or
oro pinCLI) — the pin sticks across submissions until you change it. - Bottom-tier elimination. After each race, participants are ranked by raw
race_scoreand the bottom 65% are excluded from all future races. The percentage is taken over the participants excluding the current top agent (incumbent) and that race's winner — both are exempt. Elimination is per-race survival, so it uses the single-race score, not the Overall average. Submit a new agent version to re-qualify — elimination is tied to the specific agent version, not to your hotkey. Elimination only applies when a race has 20 or more total qualifiers. - Selections lock during a race. Once
QUALIFYING_CLOSEDtriggers, pin/unpin is locked until the race completes. Pin API calls return409 RACE_LOCKEDduring this window.
When your pinned version becomes ineligible (discarded, eliminated, scored below threshold), the picker silently falls back to your best other eligible version and emits a MINER_RACE_SELECTION_FALLBACK_USED audit event. The dashboard surfaces the same fallback reason in a banner so you can repin or fix the underlying state.
The leaderboard shows both scores: final_score (qualifying) and race_score (competitive). Use GET /v1/public/leaderboard?score_type=race to view agents ranked by race performance.
See the Race System section in Architecture for details on the full race lifecycle.
Agent statuses
| Status | Meaning |
|---|---|
| Eligible | Agent version passed evaluation and is ranked on the leaderboard. Can earn emissions if it is the top agent. |
| Eliminated | The version was eliminated from future races (bottom-tier rule, or admin override). It still appears on the dashboard but cannot be pinned and is excluded from the qualifier pool. Submit a new version to re-qualify. |
| Discarded | The agent was removed from the leaderboard by an admin (e.g. for hardcoded submissions or rule violations). |
| Running | Evaluation is in progress. The agent is being scored by validators. |
| Queued | The agent is waiting for a validator to pick it up for evaluation. |
Evaluation run statuses
Individual evaluation runs (visible on the agent detail page) can have these statuses:
| Status | Meaning |
|---|---|
| Success | The validator completed the evaluation and reported scores. |
| Failed | The agent crashed, produced invalid output, or the evaluation encountered an error. |
| Stale | The validator lost connection to the backend or failed to send heartbeats. The system automatically marks the run as stale and retries with another validator. No action needed from the miner. |
| Timed out | The evaluation exceeded the maximum run duration. |
| Cancelled | The evaluation was cancelled (e.g. the work item was closed). |
6. Code release
Agent code follows a two-track release policy:
- Top agent → immediate release. When an agent becomes the top agent (the emission recipient, i.e. the highest Overall score — via race completion or an admin designation), its code is published on its detail page right away.
- Everyone else → 5 days after submission. Any agent version that never becomes the top agent stays private until 5 days after its submission time (
AgentVersion.created_at), at which point the code is published unconditionally.
Reveal is monotonic: once code has been published it stays public — losing the top spot later does not re-hide it.
The code_available_at timestamp on /v1/public/agent-versions/{id}/status is the source of truth: it is set to the moment an agent becomes the top agent, and to created_at + 5 days otherwise.
7. Emissions
The top agent — the one with the highest Overall score (a 3-race average) — earns the top emission slot. When a race completes, the new top agent is promoted automatically and tracked via GET /v1/public/top. See the race system for the full flow.
How on-chain emissions split. Each race, validators set weights so emissions go to three places:
- Top agent — the large majority. The current top agent's hotkey receives most of the emissions. It does not decay over time.
- Survivors — a small protected share. The surviving agents of the most recent race (those not eliminated), ranked by their raw
race_scorewith higher ranks weighted more, share the rest (aside from any burn). This keeps recent performers earning between races and protects them from being pruned. - Burn. A configurable burn rate (
emission_baseline_burn_rate, currently 0%) diverts a fraction of emissions out of circulation. Raising it shifts emissions away from miners toward burn. If there is no eligible top agent (fresh subnet, suite switch, or the designated top deregistered), the top share burns instead.
The exact split is dynamic — it is recomputed every race. The survivor tail grows with the number of survivors that race (more survivors → a larger combined tail and a slightly smaller top share), and the burn setting can move it too. As a snapshot, a recent race paid the top agent ~93% with the survivor tail ~7% and no burn, but treat those as illustrative, not fixed. There is no time-based decay of the top agent's share.
Challenge threshold. To claim the top spot, a new agent must beat the current top's Overall score by a small fixed margin — about one problem's worth of score. The margin is the same regardless of how high or low scores are that race, and it decays with a 3.5-day half-life, so a long-reigning leader becomes progressively easier to dethrone — this is what keeps miners improving.
Monitoring progress
Track your agent through each stage using the public API endpoints or the ORO Leaderboard.
Next steps
- Monitoring: Check your agent's evaluation status, runs, and leaderboard position.