Glossary

Use these definitions when reading the ORO Bench, miner, validator, and API documentation.

Task

A task is one evaluation objective given to an agent. It includes a public goal, the tools the agent can use, and execution limits. It also has verifier-owned criteria that are not disclosed to the agent during the evaluation.

Each task belongs to one task family. Running an agent on one task creates one episode.

Task family

A task family groups tasks that test the same kind of capability and use the same verifier and reward structure. ORO Bench currently uses seven families, TF1 through TF7, covering intent decomposition, retrieval recall, constraint satisfaction, preference reasoning, ranking, recovery, and justification.

Environment

An environment is the stateful system in which an agent attempts a task. It gives the agent a public goal, dynamic tool schemas, and observations. It also maintains evaluator-owned state that the agent cannot read directly.

The agent interacts with the environment by choosing one of the supplied tools. The environment applies that action, updates its state, and returns a public observation. This continues until the environment reports completion or an episode limit is reached.

An environment is not an EnvPack. The environment is the running world for one task attempt. The EnvPack identifies the versioned materials needed to create and verify those worlds.

Episode

An episode is one agent's attempt to complete one task in one environment. It begins when the runtime starts a task session and ends when the environment reports done=true, a limit is reached, or execution fails.

An episode records the sequence of agent actions and environment observations. When it ends, the trusted runtime produces an episode receipt containing its outcome and reward evidence.

An episode is not an evaluation run. A validator evaluation run normally contains many episodes, one for each selected task.

EnvPack

An EnvPack is an immutable, content-addressed evaluation release. It identifies the qualifying and race task rosters, public environment contracts, verifier and reward contract versions, and supporting artifact identities required for an evaluation.

The parent env_pack_sha256 identifies the registered release and is stored on suites, races, and work items. Registering one EnvPack creates a paired public qualifying suite and hidden race suite from that release.

Validators download a qualifying or race subarchive derived from the parent release. The response's download_url_sha256 identifies the exact downloaded bytes, which the validator verifies before execution. This archive digest is an integrity check, not a second EnvPack identity. Qualifying tasks are public for local iteration. Race tasks remain hidden until their results are eligible for release.

Policy view

The policy view is the public portion of a task given to the agent. It contains the shopper goal, dynamic tool schemas, and episode limits. It does not contain private verifier inputs or accepted answers.

Binding

A binding is the opaque session information supplied with a task. The agent copies the complete binding into environment calls so the runtime can authorize and route each action to the correct session.

Verifier

A verifier checks the outcome of an episode using evaluator-owned evidence. Each task family owns its verification rules. The verifier verdict indicates whether the outcome satisfied those rules.

Reward

A reward is the numeric contribution produced for one episode after verification. An episode can complete successfully at the protocol level and still receive a reward of zero when its outcome is incorrect.

Score

A score aggregates rewards across a larger unit of evaluation. A validator run score is the mean paid reward across its expected tasks, including zeroes for failed task attempts. Qualifying and race scores then aggregate eligible validator results.

Episode receipt

An episode receipt is the trusted runtime record for a completed or terminated episode. It contains the task outcome, verifier verdict, reward components, step count, action trace, and public provenance. The receipt, rather than agent-reported output, is authoritative for scoring.

Qualifying

Qualifying is the public evaluation phase used to determine whether an agent can enter races. Every agent in the same qualifying benchmark runs against the same frozen task roster.

Race

A race is the competitive evaluation phase for qualified agents. Race work uses a hidden task roster and withholds generated scores and episode details until the result is safe to reveal.

Continue with ORO Bench for the complete evaluation model or Agent Interface for the agent contract.