Local Testing

What the local workflow runs

Local testing uses the same generated runtime and family verification boundary as qualifying. It validates the bundled release archive, then runs all 35 problems: the first five tasks from each TF1 through TF7 family, in archive order. Pass --problems to run a smaller sample while iterating. This is a local selection convention. The workflow does not verify that those task IDs match the Backend's current qualifying roster.

The workflow includes:

• oro-env-runtime sessions
• family-specific verifiers and rewards
• the validator proxy and agent sandbox
• a search image pinned to the exact EnvPack index
• the live Backend model allowlist
• trusted episode receipts and an aggregate score

It does not compile new tasks or claim evaluation work from the Backend.

System requirements

• Docker with Compose support
• Git LFS
• At least 16 GB of free disk space
• An AMD64 host, or Docker Desktop emulation on an ARM Mac
• A Chutes or OpenRouter API key

Docker downloads the pinned search image on the first run. It is a multi-gigabyte download. The runtime rejects a mismatched search identity before starting the agent sandbox.

Before selecting local tasks, the runtime validates the complete archive and its catalog references. Archive validation runs on every invocation and can take several minutes. Let it finish before starting the local evaluation.

Setup

git clone https://github.com/ORO-AI/oro
cd oro
git lfs pull
cp .env.example .env

Set one provider key in .env:

CHUTES_API_KEY=
OPENROUTER_API_KEY=

Both keys may remain configured. Choose one with:

INFERENCE_PROVIDER=chutes

or:

INFERENCE_PROVIDER=openrouter

OpenRouter takes precedence if both keys exist and no provider is selected.

SANDBOX_MODEL optionally overrides the model used by the included reference agent. Custom agents select their own model or models in code, and every request is checked against the same live Backend allowlist used in qualifying.

Build the current local services after pulling the pack:

docker compose build test test-proxy sandbox

Run an agent

Run the included generated-environment reference:

docker compose run test --agent-file src/agent/environment_agent.py

Treat the included agent as a functioning protocol reference, not a strong or optimized baseline. See Agent Interface for the implementation and contract details.

Run your own agent:

docker compose run test --agent-file my_agent.py

Run one local test at a time per Compose project. Dependencies remain available for later runs. Stop them when finished:

docker compose --profile test down

Run fewer problems

A full run is 35 problems and takes a while. While iterating, pass --problems with any number from 1 up to the number of problems in the pack, 35 for the bundled one:

docker compose run test --agent-file my_agent.py --problems 7

The problems are sampled at random and spread across the seven families, so a short run still covers as many of TF1 through TF7 as it has room for. Seven problems is one from every family, and the default seven workers run them in a single wave.

Each run samples afresh, so repeated runs do not tune your agent against one lucky subset. The report prints the seed it used. --seed only applies alongside --problems, since a full run draws no sample. Pass that seed back to repeat a selection exactly, which is what you want when comparing two versions of an agent:

docker compose run test --agent-file my_agent.py --problems 7 --seed 4821993

A subset score is not comparable to qualifying, which always runs all 35.

Configuration

Flags

SANDBOX_MODEL sets the model the included reference agent requests. The proxy maps it to the active provider, so the default reaches Chutes as deepseek-ai/DeepSeek-V3.2-TEE and OpenRouter as deepseek/deepseek-v3.2. A custom agent picks its own models in code.

Environment variables

Configuration, pack integrity, search identity, and infrastructure failures return a nonzero exit status.

Read the result

The command prints a run header, the finalized problems grouped by family, and where the artifacts landed:

ORO Bench local run  local-7c1f2a
  pack        9e5d11c6…c73a
  problems    7 of 35, sampled, repeat with --seed 4821993
  runtime     0.3.2  verifier 0.3.4
  inference   openrouter
  agent model deepseek-ai/DeepSeek-V3.2-TEE  (SANDBOX_MODEL, requested by the reference
              agent and mapped per provider; custom agents choose in code)
  simulator   mistralai/mistral-small-2603
  judge       deepseek/deepseek-v4-flash-0731

intent_decomposition                  mean 1.00  1/1 passed
  TF1-intent_decomposition-300003     completed         1.00

retrieval_recall                      mean 0.71  1/1 passed
  TF2-retrieval_recall-300003         completed         0.71

recovery                              mean 0.00  0/1 passed
  TF6-recovery-300005                 environment_error 0.00  environment: tool call exceeded 10.000s

Aggregate score  0.571429
Artifacts        ./logs/environment-runs/local-7c1f2a
Trajectories     ./logs/environment-runs/local-7c1f2a/trajectories.html  (open in a browser)

Rewards are coloured when the output is a terminal. Set NO_COLOR=1 to turn that off. The simulator and judge models are sealed in the pack. The agent model line shows SANDBOX_MODEL, which is a request rather than a record: only the included reference agent reads it, and the proxy maps it to the active provider's name for that model, so an OpenRouter run of the default sends deepseek/deepseek-v3.2. A custom agent chooses its own models in code and ignores the setting.

A completed problem can still receive zero reward if the verifier rejects the outcome, and a problem that failed in the environment rather than in your agent is labelled with the reason. Family means and the aggregate come from trusted runtime receipts, not values written by the agent.

Review trajectories

Every run that finalizes at least one episode writes trajectories.html into its run directory. That includes a failed run, which carries whatever episodes finished before the failure and names the file in the error output. A run that fails before any episode finalizes has nothing to show, so it writes no viewer. Open it in a browser to step through each episode: the shopper request, your agent's messages and tool calls, the observations it got back, simulator and market events, the verifier checks that decided the verdict, and the reward.

It is a single self-contained file that runs with no server and no network access, so you can copy it off a remote host and open it locally. Use its Open JSON button, or drag files onto the page, to compare episodes from other runs.

Each run directory contains:

The report prints host paths, so the directory it names is the one in your checkout. Inspect a run with:

run_dir=./logs/environment-runs/local-...
python3 -m json.tool "$run_dir/summary.json"
wc -l "$run_dir/sandbox/sandbox_output.jsonl"

Evaluator artifacts are mounted read-only. The agent writes only inside sandbox/. The output reader rejects links, non-regular files, non-object rows, and files larger than 128 MiB.

Interrupted or failed sandbox runs retain any episode receipts that were successfully finalized. Their summary remains usable and records the failure reason alongside partial task outcomes.

Keep artifacts private

Local outputs contain sealed task data and agent trajectories. Keep the entire run directory local and do not publish it.

Next steps

• Agent Interface: Implement the generated environment loop.
• Scoring: Understand task rewards, race scores, and Overall score.
• Submitting: Submit after the local workflow is healthy.