Reinforcement learning environment

A reinforcement learning environment (RL environment) is everything outside the agent that the agent can affect. The agent sees an observation, chooses an action, and the environment applies that action, changes its own state and returns a new observation. At the end, or along the way, the environment produces a reward that says how well the agent did.

The parts of an environment

• Observations. What the agent is allowed to see: a user's request, a product page, the result of a tool call.
• Actions. What the agent is allowed to do. For a language-model agent these are usually tool calls, such as searching a catalog, adding an item to a cart or sending a message.
• State. What the environment knows and keeps track of, including facts the agent cannot read directly, such as the correct answer or hidden stock levels.
• Termination. The rule that ends an attempt: a finished order, a decline, a step limit or an error.
• Reward. The score the attempt earns. See reward hacking for what goes wrong when the reward can be met without doing the task.

Why commerce environments are different

In a commerce environment the actions have prices, stock runs out and a user's stated preferences have to be weighed against what is actually available. An agent can produce fluent text and still buy the wrong thing. Because the outcome is a concrete decision, it can be checked, which makes commerce a demanding place to train and benchmark agents.

At ORO

ORO Bench runs shopping agents in sealed environments: each task is a request from one shopper, set in a world with its own products, tools and situations, and the world's grader scores the result. The ORO Bench docs describe how those environments are built and versioned.