Research · ShoppingBench

ShoppingBench.

Alibaba’s shopping benchmark, run as a live arena, and a small agent trained on its output.

Alibaba Group created ShoppingBench (Wang et al., 2025): shopping requests grounded in real user intents, solved with product search, product detail, budget, web search and recommendation tools over more than 2.5 million real products. ORO ran a modified version on Bittensor SN15, where miners competed to build the best shopping agent, and post-trained a 4-billion-parameter model on the graded trajectories.

Benchmark by Alibaba Group (Wang et al., 2025)Research by OROPublished June 1, 2026

Qwen3-4B after ORO post-training, up from 18.0% for the base model: 42.7%
Subnet trajectories extracted: 146,831
Released training trajectories: 15,269
Solved within eight tries; 34.8% in one: 53.3%

01 · The benchmark

Built by Alibaba, run as an arena.

On SN15, validators ran miners' shopping agents on rotating ShoppingBench problems. Three features made the output useful for training:

Races: Agents competed for emissions on fresh problems, so many independent builders kept trying different strategies.
A reasoning judge: An LLM judge scored each run's reasoning, a quality signal beyond pass or fail.
Rotating problems: Near-duplicate problems were grouped, so a held-out set stays unseen.

SN15 now runs ORO Bench, ORO's own benchmark with hidden race tasks; this page covers the ShoppingBench period.

02 · Results

Trained on subnet trajectories, Qwen3-4B lands near the published synthetic-data result.

After ORO post-training, Qwen3-4B solves 42.7% of our 75 held-out problems, up from 18.0% for the base model and three more than GPT-5.5. The ShoppingBench paper reports 43.6% for the same model fine-tuned on GPT-4.1 trajectories and 48.7% with fine-tuning plus RL. Those published numbers come from the paper's own test set and harness, not runs on our problems, so the comparison is indicative. A 95% binomial interval near 42.7% on 75 problems spans about ±11 points (problems treated as independent), so the gap to GPT-5.5 is also within noise.

Figure 1Success rate: measured on ORO's 75 held-out problems, or published by ShoppingBench

Qwen3-4B

Base model

18.0%

GPT-5.5

Frontier model

38.7%

Qwen3-4B, ORO post-trained

Trained on subnet trajectories

42.7%

Qwen3-4B + SFT

Synthetic GPT-4.1 trajectories, published

43.6%

Qwen3-4B + SFT + GRPO

The published bar

48.7%

Claude Sonnet 4.6

Frontier teacher for stage 3

64.0%

Top SN15 miner

Ensemble of several models (agent_23175)

77.3%

Share of problems solved, scored strictly · higher is better →

Bars marked published are from the ShoppingBench paper (its own test set and harness); the rest were measured on ORO's 75 problems in a 7-tool harness, so cross-group comparisons are indicative only. One ORO problem is 1.3 points. The top miner, an ensemble of several models, marks what the arena reached, not a like-for-like comparison.Source: ORO paper, Table 1. Published baselines are from the ShoppingBench paper.

03 · Trajectories

From 146,831 runs to a training corpus.

We normalized every race-phase trajectory from roughly the subnet's first 40 days to one tool-call format with Trac (an annotation framework by Jarrod Barnes), then filtered hard. Most of the cut is deliberate: failed or invalid runs go, and a cap of 50 trajectories per problem keeps a few easy problems from dominating.

Figure 2Trajectories at each step

Race-phase extract

About the first 40 days

146,831

Valid, winning, complete

88,910 dropped, 63,419 failing validation

57,921

At most 50 per problem

502 problems, 35 races

18,043

Split: training

Leak-cluster guarded

15,269

Split: held out

Never trained on

2,676

The training and held-out splits of the 18,043 trajectories share no answer product: trajectories recommending a held-out answer are dropped, and near-duplicate problems stay on one side. 98 were dropped at the split.Source: the stats.json and funnel files published with both Hugging Face datasets. The day count is from the revised paper PDF on this site; arXiv v1 describes the slice differently.

04 · Training

Fine-tuning reached 42.7%; RL improved the process but not strict success.

We follow the ShoppingBench paper's fine-tune-then-RL pipeline with two substitutions, explained in our paper.

Stage 1: SFT on subnet trajectories: One epoch of LoRA fine-tuning on the filtered corpus.
Stage 2: Rejection-sampled re-SFT: Eight rollouts per training problem; the model is fine-tuned again with only perfect-score rollouts added back. Result: 34.7%.
Stage 3: SFT on frontier-teacher rollouts: Continued fine-tuning on strictly passing Claude Sonnet 4.6 rollouts, replacing the published step-wise distillation (too costly at this scale). Result: 42.7%.
Stage 4: KTO preference refinement: No overall lift (32 of 75 problems before and after); it shifted success between task types. A tighter setting regressed to 35.1%. Result: 42.7%.
Stage 5: Dr. GRPO reinforcement learning: A per-step reward grounded in the teacher’s tool calls raised the in-training rule score from 0.02 to 0.42 and cut product-ID hallucinations from 14 to 0. Strict success did not move.

Figure 3Held-out success through the fine-tuning stages

After stage 2 (rejection_v2)

34.7%

Teacher SFT (opd_sonnet)

34.7%

Rebalanced teacher corpus (opd_rebal)

40.0%

Final SFT stack (opd_renderers)

42.7%

Share of the 75 held-out problems solved · higher is better →

Rebalancing the teacher corpus across task types, then fixing a loss-masking bug in fine-tuning, gave the last two gains.Source: ORO paper, Table 1.

Within eight tries the model solves 53.3% of held-out problems, against 34.8% in one: the gap RL is meant to close.

Figure 4One try against eight

pass@1

One try per problem

34.8%

pass@8

Any of eight tries

53.3%

Final fine-tuned checkpoint, same held-out problems. The probe samples its tries, so its one-try rate is below the 42.7% evaluation score; the paper reports both.Source: ORO paper, Section 7.

05 · Limitations

What these results don’t show.

Most subnet output wasn't usable. Top miners use the model only to classify and narrate, with search and ranking in plain code. Such runs dominate daily output but don't teach tool use, so we trained on the smaller agentic share. The paper treats the result as a lower bound on what the arena can produce.
The harness differs from the paper's. ORO's 7-tool harness and prompts differ from the paper's own setup. We cross-checked part of the difference; comparisons with published numbers still carry it.
The quality filter is a model's judgment. The reasoning score gating the corpus comes from an LLM judge, not a gold label, and inherits its errors.
RL didn't reach the published bar. Dr. GRPO left strict success at the fine-tuning level, short of the paper's 48.7%.
The held-out set is small. At 75 problems, one problem moves a score by 1.3 points.

06 · Data and citation

Everything is released.

Cite Alibaba's paper for the benchmark, and ours for the corpus, models or results.

@article{wang2025shoppingbench,
  title   = {ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents},
  author  = {Wang, Jiangyuan and Xiao, Kejun and Sun, Qi and Zhao, Huaipeng and Luo, Tao and Zhang, Jian Dong and Zeng, Xiaoyi},
  journal = {arXiv preprint arXiv:2508.04266},
  year    = {2025}
}

@article{oro2026trajectory,
  title   = {Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces},
  author  = {{ORO}},
  journal = {arXiv preprint arXiv:2606.10064},
  year    = {2026}
}