ICCA Bench.

Language-model agents are increasingly deployed in long-horizon operational settings, where the effects of their decisions are delayed, unlabeled, and confounded with exogenous noise. Whether an agent can learn from such consequences, and reuse that experience when conditions change, is not measured by existing benchmarks, which largely provide immediate and verifiable feedback.

We introduce ICCA Bench. Its first environment is a simulation of a produce-wholesale buying desk calibrated to public retail, price, and freight data, in which an agent makes purchasing decisions over a 90-day quarter. Each task hides one consequence and pairs the world with a counterfactual twin that is identical except for that consequence, with the same initial state and random draws. The pairing yields a learning signal that separates adaptation to the consequence from chance and from generally good play. Tasks are admitted only if the consequence is material, learnable from the agent's own records, and not satisfied by a fixed habit. In interim results on the 12 active tasks, Claude Opus 5.5 averages −4.14% of baseline revenue against a fixed buyer that ignores the consequence and Grok 4.7 −8.22%; GPT-6 Astra averages +1.40% on its 10 completed active tasks. Trace review finds profitable standing policies, evidence-linked responses that lose money, and one blind-reviewed supplier response that also beats the fixed buyer. These are early observations from one run per condition, not a ranking of models, and they do not yet establish whether agents learn from delayed consequences: the contribution of this report is the question and the way to measure it. We release the environment, the 18 public tasks, and reward-only training streams.

Setting. Agents are being assigned multi-week operational roles: procurement, inventory, account management. Feedback in these roles is delayed, unlabeled, and confounded, so attributing an outcome to an earlier decision is part of the task. For example, business customers reduce orders after repeated short deliveries (Craig, DeHoratius and Raman, 2016).

Approach. Paired counterfactual worlds isolate the effect of each consequence. Admission criteria require each task to be material, learnable, and not gameable. The grader scores value created, and reports a behavioural criterion, a transfer measure, and a per-model noise floor beside it. Blind trace review assesses the four stages.

Findings so far. Value created varies widely between models. Profitable standing policies earn money without a learned response, and responses that follow the evidence can still lose to the fixed buyer. With one run per condition, learning is reported as an observation; these runs do not yet establish it.

Formalization. A partially observable, stochastic decision process over 90 epochs (simulated mornings).

State

Inventory by lot and age, open orders, cash and receivables, and latent supplier and customer parameters.

Actions

Purchase orders (product, supplier, quantity) and standing rules for covering shortfalls at the local market.

Observations

The agent's own records (orders, receipts, inspections, claims, sales, customer messages), queried through SQL, plus each morning's supplier offers.

Transition

Fixed operating rules run the rest of each day: delivery and inspection, filling account orders, walk-in sales, spoilage, claims, and billing.

Business family. Businesses share one structure and differ in demand, prices, supplier reliability, and customer sensitivity. Each has several suppliers with distinct price and reliability profiles, 6 business accounts, and walk-in demand.

Agent interface. A fixed system prompt states the operating rules a practitioner would know. Each epoch has a tool-call budget and persistent memory files. The agent controls purchasing only. It is never told which mechanisms exist, whether one is active, or how it is graded.

Reproducibility. Worlds are seeded, releases are frozen by hash, and every run can be replayed from its action log.

Tasks and paired worlds. A task pairs a business with one hidden mechanism. Its treated world and its twin share the initial observation and all random draws, and differ only in that mechanism.

Census. 504 scripted runs. 2 of 6 candidate mechanisms were admitted, giving 12 tasks plus 6 null tasks: 18 public tasks. Quality, shelf-life, demand, and walk-in mechanisms were not material and are reported as negative results.

Splits. Public: businesses 500–505. Training: 506–537, reward only. Held-out: 153 unopened stores, kept private.

Reward. Each run is scored on the net business value it creates over the quarter: cash, receivables and stock at liquidation value, less what the business owes and the margin already lost from customers, graded independently from the event log. Its reward is that value minus the value created by a fixed buyer that ignores the hidden consequence, in the same world, expressed as a share of the business's baseline revenue so that businesses of different sizes compare. Value created is the primary outcome.

Learning signal. A model's reward in the world with the hidden consequence minus its reward in the twin without it. The two worlds start identically and share their random draws, so the difference reflects how the model handled the consequence rather than luck or general skill. It is read against the damage the same consequence does to the fixed buyer.

Learned. A task counts as learned when the learning signal is at least 0.25% of baseline revenue and the model's behaviour changed after the consequence began: in some 14-session window after onset, its supplier choice, local cover or order quantity moved at least 0.10 from its level before onset in the direction the consequence rewards, and at least 0.10 more in that direction than in the twin. The level before onset leaves out the first 2 days, which are setup; the change must not already be in place over the last 3 sessions before onset; and local purchases count only under cover instructions set after onset. Learned is reported beside value as an observation; it never changes the reward.

Transfer. In a later quarter with a supply disruption, the learning signal when the model starts with its own archive of records and notes, minus the learning signal from a cold start.

Noise. Each business's null task runs the same model twice in identical worlds. The gap between those two runs is the model's run-to-run noise on that business, and a learning signal whose size, ignoring its sign, is smaller than twice that gap is reported as within noise.

3 frontier models ran the 18 public tasks, each buying for a business over a 90-day quarter. These are interim results: GPT-6 Astra has 3 of its 18 tasks still running; Transfer is still running for every model. Missing runs are pending, never counted as zero.

Do agents create value? Only GPT-6 Astra beats the fixed buyer on average. By task, Claude Opus 5.5 beats it on 3 of 12, GPT-6 Astra beats it on 6 of 10, Grok 4.7 beats it on 0 of 12 (Table 2, Figure 3).

Do agents learn from the hidden consequence? 4 of 34 completed active pairs meet the learned criterion (Claude Opus 5.5 0, GPT-6 Astra 3, Grok 4.7 1). We report this as an observation: one run per condition cannot separate a model's learning from run-to-run variation, and a change in behaviour and the value created do not always agree.

Where does the loop break? Trace review of selected runs, chosen to illustrate rather than sampled, shows three patterns: a standing protective policy that earns money without any learned change; responses that notice the consequence, attribute it and act, yet still trail the fixed buyer; and one supplier response, confirmed by blind reviewers, that adapts and beats the fixed buyer. Reviewers read accepted actions, public queries and notes, never private model reasoning.

Does experience transfer? Transfer is still running. It has finished for 2 tasks, on one business, with mixed results, which cannot yet say whether experience helps.

Summary. ICCA Bench's first environment is an audited simulation with real economic consequences, and a measurement designed to separate value from learning. The interim results show competent prevention earning money without a learned response, and agents that diagnose and act on delayed evidence but still lose. A completed three-model evaluation of Transfer, and a demonstration of reliable learning, are still to come.

Code and datasets verified on October 22, 2026. All are private to the team until release. Training streams: 12 of 32 businesses published, verified by re-download.

@misc{oro2026iccabench,
  title  = {ICCA Bench: Measuring Whether Agents Learn from the Delayed Consequences of Their Decisions},
  author = {Barnes, Jarrod and Schilbe, Seth and Bansal, Shardul},
  year   = {2026},
  note   = {Technical report. Environment: WholesaleBench v1, retail-v2-dev at 6f75291},
  url    = {https://oroagents.com/benchmarks/icca-bench}
}