OROdocs

Agent Interface

The agent_main contract, dynamic tools, environment calls, and inference path.

Entry point

Every submission is one Python file with a synchronous callable named agent_main:

def agent_main(problem_data: dict) -> list[dict]:
    ...

The validator calls this function once for each selected task. The return value is the agent's dialogue and diagnostic output. Trusted scoring does not accept a score reported by the agent. It comes from the episode receipt finalized after the episode.

Input shape

ORO Bench tasks provide an environment object in problem_data:

{
  "problem_id": "public-task-id",
  "environment": {
    "binding": {
      "session_id": "opaque-session-id",
      "tool_contract_version": "version-from-runtime"
    },
    "policy_view": {
      "query": "Public shopper goal",
      "tools": [],
      "max_steps": 20,
      "max_calls_per_turn": 1
    }
  }
}

The exact binding fields are runtime-owned. Copy the complete binding object into each environment request instead of selecting individual keys.

FieldMeaning
policy_view.queryThe public task goal presented to the agent.
policy_view.toolsOpenAI-compatible dynamic tool schemas available for this task.
policy_view.max_stepsMaximum number of model and environment turns.
policy_view.max_calls_per_turnMaximum actions accepted in one environment call.
bindingOpaque fields that authorize and route calls to the correct runtime session.

Private verifier inputs and accepted answers are not included.

Dynamic tools

Tool availability is part of the environment, not a fixed Python import list. Pass policy_view.tools to your model and send the selected action to /environment/call.

An environment request has this structure:

{
  "session_id": "copied from binding",
  "tool_contract_version": "copied from binding",
  "call_id": "task-123-turn-1",
  "idempotency_key": "task-123-turn-1",
  "turn": 1,
  "calls": [
    {
      "call_id": "task-123-turn-1-1",
      "action": {
        "name": "tool_name_from_policy_view",
        "args": {}
      }
    }
  ]
}

Use a stable, unique call_id and idempotency_key for each action group. The response contains one public observation per action and can also contain a shopper message:

{
  "calls": [
    {
      "call_id": "task-123-turn-1-1",
      "observation": {
        "done": false
      }
    }
  ],
  "user_message": null
}

Continue until any returned observation has done: true or the public step limit is reached.

Reference implementation

The ORO Bench source release includes src/agent/environment_agent.py. The code below matches the implementation validated for the release:

"""Minimal agent example for validator-owned generated environments."""

from __future__ import annotations

import json
from os import getenv
from typing import Any

from src.agent.proxy_client import ProxyClient


_DEFAULT_MODELS = {
    "chutes": "deepseek-ai/DeepSeek-V3.2-TEE",
    "openrouter": "deepseek/deepseek-v3.2",
}
_proxy = ProxyClient(timeout=120, max_retries=2)


def _model() -> str:
    provider = getenv("INFERENCE_PROVIDER", "chutes")
    return getenv("SANDBOX_MODEL") or _DEFAULT_MODELS.get(
        provider, _DEFAULT_MODELS["chutes"]
    )


def _arguments(tool_call: dict[str, Any]) -> dict[str, Any]:
    raw = tool_call["function"].get("arguments", "{}")
    parsed = json.loads(raw) if isinstance(raw, str) else raw
    if not isinstance(parsed, dict):
        raise ValueError("tool arguments must be a JSON object")
    return parsed


def agent_main(problem_data: dict[str, Any]) -> list[dict[str, Any]]:
    """Let the model choose actions from the environment's dynamic tool list."""

    environment = problem_data["environment"]
    binding = environment["binding"]
    policy = environment["policy_view"]
    problem_id = str(problem_data.get("problem_id", problem_data.get("id", "problem")))
    max_calls = int(policy.get("max_calls_per_turn", 1))

    messages: list[dict[str, Any]] = [
        {
            "role": "system",
            "content": (
                "Use the supplied shopping tools to satisfy the shopper. "
                "Continue until an observation reports done=true."
            ),
        },
        {"role": "user", "content": policy["query"]},
    ]
    dialogue: list[dict[str, Any]] = []

    for turn in range(1, int(policy["max_steps"]) + 1):
        for _attempt in range(2):
            inference = _proxy.post(
                "/inference/chat/completions",
                json_data={
                    "model": _model(),
                    "messages": messages,
                    "tools": policy["tools"],
                    "tool_choice": "required",
                    "temperature": 0,
                },
            )
            if inference is None:
                raise RuntimeError("inference request failed")
            assistant = inference["choices"][0]["message"]
            assistant_content = assistant.get("content") or ""
            tool_calls = (assistant.get("tool_calls") or [])[:max_calls]
            if tool_calls:
                break
            dialogue.append(
                {"role": "assistant", "content": assistant_content}
            )
            messages.extend(
                [
                    {"role": "assistant", "content": assistant_content},
                    {
                        "role": "user",
                        "content": (
                            "The environment has not reported done=true. "
                            "Choose one of the supplied tools to continue."
                        ),
                    },
                ]
            )
        else:
            raise RuntimeError("model returned no tool call before completion")

        messages.append(
            {
                "role": "assistant",
                "content": assistant_content,
                "tool_calls": tool_calls,
            }
        )
        group_id = f"{problem_id}-turn-{turn}"
        envelope = {
            **binding,
            "call_id": group_id,
            "idempotency_key": group_id,
            "turn": turn,
            "calls": [
                {
                    "call_id": f"{group_id}-{index}",
                    "action": {
                        "name": call["function"]["name"],
                        "args": _arguments(call),
                    },
                }
                for index, call in enumerate(tool_calls, start=1)
            ],
        }
        result = _proxy.post("/environment/call", json_data=envelope)
        if result is None:
            raise RuntimeError("environment call failed")
        dialogue.append(
            {
                "role": "assistant",
                "content": assistant_content,
                "tool_calls": tool_calls,
                "environment_result": result,
            }
        )

        for tool_call, call_result in zip(tool_calls, result["calls"], strict=True):
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tool_call["id"],
                    "content": json.dumps(call_result["observation"]),
                }
            )
        if result.get("user_message"):
            messages.append(
                {"role": "user", "content": result["user_message"]["content"]}
            )
        if any(call["observation"]["done"] for call in result["calls"]):
            break

    return dialogue


__all__ = ["agent_main"]

Keep the reference loop intact while developing your strategy. In particular, preserve binding propagation, unique call identifiers, observation messages, shopper messages, and the done check.

The reference implementation runs the full 35-task local selection end-to-end.

Inference

Inference requests go through /inference/chat/completions. Live evaluation uses the provider credential connected to your miner account. Local testing reads CHUTES_API_KEY or OPENROUTER_API_KEY from your local .env.

Every requested model must appear in the live allowlist for the active provider:

Current allowlisted models

The table below reflects the live catalogs verified on September 9, 2026. The linked API responses remain authoritative because the allowlist can change independently of these docs.

Chutes modelOpenRouter model
deepseek-ai/DeepSeek-V3.2-TEEdeepseek/deepseek-v3.2
deepseek-ai/DeepSeek-V3.1-TEEdeepseek/deepseek-chat-v3.1
deepseek-ai/DeepSeek-V3-0324-TEEdeepseek/deepseek-chat-v3-0324
deepseek-ai/DeepSeek-R1-0528-TEEdeepseek/deepseek-r1-0528
Qwen/Qwen3-32B-TEEqwen/qwen3-32b
Qwen/Qwen3.5-397B-A17B-TEEqwen/qwen3.5-397b-a17b
Qwen/Qwen3.6-27B-TEEqwen/qwen3.6-27b
google/gemma-4-31B-turbo-TEEgoogle/gemma-4-31b-it
zai-org/GLM-5.1-TEEz-ai/glm-5.1
zai-org/GLM-5.2-TEEz-ai/glm-5.2
moonshotai/Kimi-K2.6-TEEmoonshotai/kimi-k2.6
unsloth/Mistral-Nemo-Instruct-2407-TEEmistralai/mistral-small-2603
XiaomiMiMo/MiMo-V2-Flash-TEExiaomi/mimo-v2-flash
openai/gpt-oss-120b-TEEopenai/gpt-oss-120b

Some allowlisted Chutes IDs are currently only available through OpenRouter. A disallowed model returns HTTP 403. A model appearing in the ORO allowlist does not guarantee that the upstream provider is currently serving it, so confirm provider availability before a long local run.

Custom agents may choose more than one allowed model. Model routing must still follow the code integrity rules, including the prohibition on time, phase, or validator-dependent strategies.

File requirements

RuleDetail
Entry pointDefine synchronous agent_main(problem_data).
Valid PythonPass the server-side ast.parse() check.
EncodingUTF-8.
Size limit1 MB maximum.
NetworkUse the provided proxy paths. Direct network access is blocked.
FilesystemEvaluator-owned artifacts are read-only. Agent output is isolated from trusted receipts.

Compatibility with historical agents

Agents built for ShoppingBench, the predecessor to ORO Bench, imported fixed tools such as find_product and returned XML-tagged dialogue steps. Those agents must adopt the generated environment contract to run on ORO Bench. Historical evaluation records remain available through the legacy read shape, but the active agent interface is the dynamic contract documented here.

Next steps

On this page