Local Testing

Run the production-equivalent ORO Bench qualifying workflow locally with Docker.

What the local workflow runs

Local testing uses the same generated runtime and family verification boundary as qualifying. It validates the bundled release archive, then selects the first five tasks from each TF1 through TF7 family in archive order. This is a local selection convention. The workflow does not verify that those task IDs match the Backend's current qualifying roster.

The workflow includes:

oro-env-runtime sessions
family-specific verifiers and rewards
the validator proxy and agent sandbox
a search image pinned to the exact EnvPack index
the live Backend model allowlist
trusted episode receipts and an aggregate score

It does not compile new tasks or claim evaluation work from the Backend.

Release identity

The public repository carries the exact release archive at data/local-test/env-pack.tar.gz through Git LFS.

Contract	Value
Release	R6
Bundled archive size	21.2 MB compressed
Bundled archive SHA-256	`1faa14eff9a9fb5ca8ef6bb5c09205341cc1be889bd7df5e48ab396a675381f3`
Python package	`oro-env-runtime` 0.2.7
Runtime contract	0.3.1
Tool contract	v5
Verifier contract	0.3.3
Release tasks	105
Local selection	35 tasks, the first five per family in archive order
Search image download	About 8.65 GB

Package and contract versions are different identifiers. The Python package implements the runtime, while the archive seals the contract versions it requires. LOCAL_ENV_PACK_SHA256 verifies the bundled archive bytes. It does not establish that the locally selected task IDs equal the active network suite.

Archive validation runs on every invocation and can take several minutes. Let it finish before starting the local evaluation.

System requirements

Docker with Compose support
Git LFS
At least 16 GB of free disk space
An AMD64 host, or Docker Desktop emulation on an ARM Mac
A Chutes or OpenRouter API key

Docker downloads the pinned search image on the first run. The runtime rejects a mismatched search identity before starting the agent sandbox. See Release identity for the current download size.

Before selecting local tasks, the runtime validates the complete archive and its catalog references.

Setup

git clone https://github.com/ORO-AI/oro
cd oro
git lfs pull
cp .env.example .env

Set one provider key in .env:

CHUTES_API_KEY=
OPENROUTER_API_KEY=

Both keys may remain configured. Choose one with:

INFERENCE_PROVIDER=chutes

or:

INFERENCE_PROVIDER=openrouter

OpenRouter takes precedence if both keys exist and no provider is selected.

SANDBOX_MODEL optionally overrides the model used by the included reference agent. Custom agents select their own model or models in code, and every request is checked against the same live Backend allowlist used in qualifying.

Build the current local services after pulling the pack:

docker compose build test test-proxy sandbox

Run an agent

Run the included generated-environment reference:

docker compose run test --agent-file src/agent/environment_agent.py

Treat the included agent as a functioning protocol reference, not a strong or optimized baseline. See Agent Interface for the implementation and contract details.

Run your own agent:

docker compose run test --agent-file my_agent.py

Run one local test at a time per Compose project. Dependencies remain available for later runs. Stop them when finished:

docker compose --profile test down

Configuration

Variable	Default	Purpose
`INFERENCE_PROVIDER`	OpenRouter when both keys exist	Select `chutes` or `openrouter`.
`SANDBOX_MODEL`	`deepseek-ai/DeepSeek-V3.2-TEE`	Override the included reference agent's model.
`BACKEND_URL`	`https://api.oroagents.com`	Source of the live model allowlist.
`LOCAL_MAX_WORKERS`	`7`	Maximum concurrent problem workers.
`LOCAL_TIMEOUT`	`1800`	Sandbox execution budget in seconds. Pack validation and session setup happen before this timer.
`LOCAL_ENV_PACK_PATH`	Bundled release pack	Use another pack available inside `/workspace`.
`LOCAL_ENV_PACK_SHA256`	Bundled release digest	Pin the expected digest for an override pack.

Configuration, pack integrity, search identity, and infrastructure failures return a nonzero exit status.

Read the result

The command prints 35 finalized task rows followed by the aggregate and artifact location:

intent_decomposition: completed, reward=...
retrieval_recall: completed, reward=...
... 33 more task rows ...
Aggregate score: ...
Artifacts: /app/logs/environment-runs/local-...

A completed task can still receive zero reward if the verifier rejects the outcome. Family rewards and the aggregate come from trusted runtime receipts, not values written by the agent.

Each run directory contains:

File	Contents
`summary.json`	Bundled archive digest, local task roster, per-task and per-family rewards, runtime error classification, and aggregate score.
`sandbox/sandbox_output.jsonl`	Untrusted agent trajectory output.
`episode_results.jsonl`	Finalized runtime receipts with verifier verdicts, call traces, ledgers, and provenance.
`environment_sessions.json`	Runtime session inputs.
`problems.jsonl`	Selected task inputs.

The printed /app/logs/ path maps to ./logs/ on the host. Inspect a run with:

run_dir=./logs/environment-runs/local-...
python3 -m json.tool "$run_dir/summary.json"
wc -l "$run_dir/sandbox/sandbox_output.jsonl"

Evaluator artifacts are mounted read-only. The agent writes only inside sandbox/. The output reader rejects links, non-regular files, non-object rows, and files larger than 128 MiB.

Interrupted or failed sandbox runs retain any episode receipts that were successfully finalized. Their summary remains usable and records the failure reason alongside partial task outcomes.

Keep artifacts private

Local outputs contain sealed task data and agent trajectories. Keep the entire run directory local and do not publish it.

Next steps

Agent Interface: Implement the generated environment loop.
Scoring: Understand task rewards, race scores, and Overall score.
Submitting: Submit after the local workflow is healthy.

On this page