Local Testing

What the local workflow runs

Local testing runs your agent on the public practice world, with the same runtime and validator path as qualifying and races. The practice world is built the same way as the worlds agents are evaluated on, but from a separate set of products, tools and situations. Practising there builds skills that carry over; its answers do not.

The repository bundles a signed practice delivery, and the runner refuses any other kind of pack. A practice delivery includes its answers, so each task is graded on your machine exactly as a validator would grade it. By default the runner runs every task in the delivery; pass --problems to sample fewer while iterating. The practice world is not the network's current qualifying world, and the workflow does not verify its tasks against qualifying.

The workflow includes:

• oro-env-runtime sessions, one step per tool call
• the world's own grader, scores and failure categories
• the validator proxy and agent sandbox
• the search service, loaded with the world's own index
• the live Backend model allowlist

It does not compile new worlds or claim evaluation work from the Backend.

System requirements

• Docker with Compose support
• Git LFS
• At least 16 GB of free disk space
• An AMD64 host, or Docker Desktop emulation on an ARM Mac
• An OpenRouter API key

Docker downloads the validator service images on the first run.

Before selecting local tasks, the runtime checks the delivery's signature and every file it lists, then loads the world. Archive validation runs on every invocation and can take several minutes. Let it finish before starting the local evaluation.

Setup

git clone https://github.com/ORO-AI/oro
cd oro
git lfs pull
cp .env.example .env

Set your OpenRouter key in .env:

OPENROUTER_API_KEY=

Leave INFERENCE_PROVIDER unset or set it to openrouter.

SANDBOX_MODEL optionally overrides the model used by the included reference agent. Custom agents select their own model or models in code, and every request is checked against the same live Backend allowlist used in qualifying.

Build the current local services after pulling the practice delivery:

docker compose build test test-proxy sandbox

Run an agent

Run the included reference agent:

docker compose run test --agent-file src/agent/environment_agent.py

Treat the included agent as a functioning protocol reference, not a strong or optimized baseline. See Agent Interface for the implementation and contract details.

Run your own agent:

docker compose run test --agent-file my_agent.py

Run one local test at a time per Compose project. Dependencies remain available for later runs. Stop them when finished:

docker compose --profile test down

Run fewer problems

The default run is every task in the practice delivery. To run a sample instead, pass --problems with any number from 1 up to the number of problems in the delivery. A small sample is the quickest check while iterating:

docker compose run test --agent-file my_agent.py --problems 2

The problems are sampled at random from the whole delivery, and up to seven run at once (LOCAL_MAX_WORKERS).

Each run samples afresh, so repeated runs do not tune your agent against one lucky subset. The report prints the seed it used. --seed only applies alongside --problems, since a full run draws no sample. Pass that seed back to repeat a selection exactly, which is what you want when comparing two versions of an agent:

docker compose run test --agent-file my_agent.py --problems 2 --seed 4821993

A local score is not comparable to qualifying, which runs a different world.

Configuration

Flags

SANDBOX_MODEL sets the model the included reference agent requests. The proxy maps the default to OpenRouter's deepseek/deepseek-v3.2. A custom agent picks its own models in code.

Environment variables

Configuration, pack integrity, and infrastructure failures return a nonzero exit status.

Read the result

The command prints a run header, then the finished problems grouped by failure category with each one's score, and where the artifacts landed. Full correct orders are grouped under correct.

ORO Bench local run  local-7c1f2a
  pack        5b0e93a1…41f2
  world       practice-world
  problems    all 5
  runtime     4.0.0
  inference   openrouter
  agent       my_agent.py  sha256 3f9c1d7b0000…
  agent model deepseek-ai/DeepSeek-V3.2-TEE  (SANDBOX_MODEL, requested by the reference
              agent and mapped per provider; custom agents choose in code)

correct  2 task(s)
  task-01  1.00
  task-02  1.00

did not finish  1 task(s)
  task-05  0.00

not the best option  1 task(s)
  task-03  0.00

request not met  1 task(s)
  task-04  0.00

Aggregate score  0.400000
Artifacts        ./logs/environment-runs/local-7c1f2a
Trajectories     ./logs/environment-runs/local-7c1f2a/trajectories.html  (open in a browser)

Each task scores 1.00 or 0.00, and the group it appears under names its failure category. Scores are coloured when the output is a terminal. Set NO_COLOR=1 to turn that off. The runtime line is the oro-env-runtime version running the world. The agent model line shows SANDBOX_MODEL, which is a request rather than a record: only the included reference agent reads it, and the proxy maps it to the active provider's name for that model, so an OpenRouter run of the default sends deepseek/deepseek-v3.2. A custom agent chooses its own models in code and ignores the setting.

A completed problem can still score zero, and a problem that failed in the environment rather than in your agent is labelled with the reason. Per-task scores and the aggregate come from the world's grader, not values written by the agent.

Review trajectories

Every run that finalizes at least one episode writes trajectories.html into its run directory. That includes a failed run, which carries whatever episodes finished before the failure and names the file in the error output. A run that fails before any episode finalizes has nothing to show, so it writes no viewer. Open it in a browser to step through each episode: the shopper request, your agent's tool calls, the observations it got back, including the shopper's replies, and the score and failure category. On the network, an episode's feedback shows the same transcript with its score and failure category.

It is a single self-contained file that runs with no server and no network access, so you can copy it off a remote host and open it locally. Use its Open JSON button, or drag files onto the page, to compare episodes from other runs.

Each run directory contains:

The report prints host paths, so the directory it names is the one in your checkout. Inspect a run with:

run_dir=./logs/environment-runs/local-...
python3 -m json.tool "$run_dir/summary.json"
wc -l "$run_dir/sandbox/sandbox_output.jsonl"

Evaluator artifacts are mounted read-only. The agent writes only inside sandbox/. The output reader rejects links, non-regular files, non-object rows, and files larger than 128 MiB.

Interrupted or failed sandbox runs retain any episodes that were successfully graded. Their summary remains usable and records the failure reason alongside partial task outcomes.

Keep artifacts private

Local outputs contain your agent's trajectories. Keep the run directory local and do not publish it.

Next steps

• Agent Interface: Implement the environment loop.
• Scoring: Understand task scores, race scores, and Overall score.
• Submitting: Submit after the local workflow is healthy.