Every SMB Bench run

In each run, one model has the buying desk of a simulated produce wholesaler for a 90-day quarter. Open any of the 859 runs, from 16 models, to see how its profit tracked the fixed buying rule, what it ordered and what it wrote in its notes each morning.

Made more than the fixed buying rule in the same quarter.

OpenAIGPT-6 Astra

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$16.2k$15.5k$18.2k
2−$2.6k$2.7k$3.4k$2.1k
3$616$5.7k$5.2k$5k
4$8.1k$6.7k$5.9k$6.1k
5$18.3k$19.6k$9.9k$22.8k
6$557$5.3k$4.9k$3.4k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k$11.3k$3.9k$11.2k
2−$6.1k−$1.3k−$121−$1.2k
3−$3.7k$3.3k$2.9k$3.9k
4$5.7k$1.1k$4.7k$3.1k
5$8.9k$15.2k$15.3k$15.6k
6−$5.1k−$1.1k−$6.3k−$4.1k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$15.5k$20.6k$20.4k
2−$598$3.9k$3.5k$2.7k
3$5.1k$6.3k$5.9k$5.8k
4$9.1k$8.9k$7.3k$8.5k
5$23.5k$15.5k$19.7k$22.6k
6$4.6k$5k$6.2k$4.5k

GoogleGemini 4 Carbon

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$20.2k$17.6k$19.9k
2−$2.6k$3k$2.2k$3k
3$616$4.8k$4.8k$4.8k
4$8.1k$7.5k$8.1k$7.3k
5$18.3k$22.6k$18.4k$23.5k
6$557$3.3k$1.3k$2.5k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k$12.8k$9.9k$13.5k
2−$6.1k−$386−$625−$846
3−$3.7k$3.7k$3.1k$3k
4$5.7k$5.2k$2.5k$2.1k
5$8.9k$7.9k$6.9k$9.5k
6−$5.1k−$3.4k−$1.1k−$665
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$14.5k$19.9k$14.5k
2−$598$2.6k$3.1k$2.8k
3$5.1k$5.3k$4.3k$5.8k
4$9.1k$8k$9.5k$9.7k
5$23.5k$20.3k$18.9k–
6$4.6k$2.9k$3.6k$2.9k

AnthropicClaude Fable 5.2

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$12k$12.6k$14.5k
2−$2.6k$1.6k$1.1k−$52
3$616$4k$3.6k$1.9k
4$8.1k$8k$6.3k$8.2k
5$18.3k$8.3k$26.1k$8.1k
6$557$332−$1.3k$2.2k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k$11.4k$11.8k$14.7k
2−$6.1k−$2.5k−$3.2k−$2.1k
3−$3.7k$1k$800−$603
4$5.7k$3.3k$8.2k$803
5$8.9k$5.2k$2.2k$2.9k
6−$5.1k−$1.4k$509−$3.6k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$20.4k$22.1k$18k
2−$598$3.8k$4.2k$4.1k
3$5.1k$6.6k$5.8k$5.6k
4$9.1k$11.7k$10.1k$12.5k
5$23.5k$28.8k$28.4k$24k
6$4.6k$6.5k$6.7k$6.6k

AnthropicClaude Fable 5.1

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$13k$12k$14.2k
2−$2.6k$856−$281$312
3$616$1.3k$1.1k$2.9k
4$8.1k$7.7k$8.9k$5.5k
5$18.3k$18.1k$10.8k−$1.8k
6$557−$2.2k−$682$582
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k$11.3k$6k$5.8k
2−$6.1k−$1.6k−$2.6k−$3.4k
3−$3.7k$1.5k−$1.4k−$806
4$5.7k$1.4k$3k$7k
5$8.9k$10.1k−$3.5k$3.3k
6−$5.1k−$7.1k−$984−$2.9k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$14.4k$16.2k$17.5k
2−$598$779$1.4k$2k
3$5.1k$2.9k$5.6k$2.8k
4$9.1k$8.8k$10.5k$11.2k
5$23.5k$23.4k$24.5k$23.6k
6$4.6k$5.3k$4$5k

OpenAIGPT-6.1 Sol

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$12.9k$15.1k$11.7k
2−$2.6k$2.7k$3k$2.5k
3$616$5k−$517$4.2k
4$8.1k$6.6k$5.8k$5.9k
5$18.3k−$1.7k$14.3k$3.7k
6$557−$4.8k−$1.2k−$2.6k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$11k−$1.5k$2.9k
2−$6.1k−$2k−$1.9k−$2k
3−$3.7k$2.6k$1.5k−$2k
4$5.7k$2.2k−$144−$4k
5$8.9k$16.1k$14.8k$23.2k
6−$5.1k−$5.8k−$8.7k−$1.1k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$18.4k$13.5k$17.4k
2−$598$2.9k$3.7k$3.6k
3$5.1k$5.4k$5.7k$6.4k
4$9.1k$8.8k$6.6k$7.3k
5$23.5k$25.9k$22.8k$22k
6$4.6k$5.4k$4.8k$5.5k

GoogleGemini 4 Hydrogen

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$9.1k$14.2k$10.6k
2−$2.6k−$846−$2.4k−$210
3$616$4.5k$1.7k$4.3k
4$8.1k$2.3k$7.8k−$329
5$18.3k$3.6k$4.7k−$1.4k
6$557$1.1k−$12.3k−$3.8k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k$763−$3.8k$9.2k
2−$6.1k−$804−$2k−$443
3−$3.7k−$3.7k−$9.4k−$7.7k
4$5.7k$3k$877−$3.1k
5$8.9k−$14.2k–−$8.8k
6−$5.1k−$7.4k−$9.9k−$3.6k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$16.6k$17.8k$17.1k
2−$598$3.8k$3.5k$3.1k
3$5.1k$5.6k$5k$6k
4$9.1k$6.4k$8.3k–
5$23.5k$23.3k$12.4k$21.4k
6$4.6k$2.6k$3.1k$4.8k

GoogleGemini 4 Argon

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$9.3k$4.1k$19.7k
2−$2.6k−$1.3k$1.1k$875
3$616$3.5k$2.5k$3k
4$8.1k$4k$7.8k$3.3k
5$18.3k$15.4k−$8.6k−$14.1k
6$557$2.7k−$9.2k$1k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$4.4k$15.4k$5.1k
2−$6.1k−$3.6k−$9.6k−$11.4k
3−$3.7k−$1.1k−$2k−$2.7k
4$5.7k−$1k−$4.6k$1.6k
5$8.9k−$5.9k$2k−$4.3k
6−$5.1k−$6.4k−$1.6k−$12.2k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$13.4k$19.1k$11.5k
2−$598$2k$2.3k$2.6k
3$5.1k$3.1k$3.3k$3.8k
4$9.1k$3.2k$2.8k$5.5k
5$23.5k$4.4k$17.1k−$10.2k
6$4.6k$2.4k−$5.7k$2.1k

AnthropicClaude Opus 5.5

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$4.9k$17.9k$14k
2−$2.6k−$4.3k−$155−$4k
3$616−$3.6k−$4.9k−$154
4$8.1k$2.2k−$131−$3.9k
5$18.3k−$1k$4.5k−$757
6$557$1.5k−$2k$1.2k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$6.7k−$4.1k−$3.6k
2−$6.1k−$8.9k−$5.4k−$8.2k
3−$3.7k−$4.6k−$7.3k−$8.5k
4$5.7k−$9.5k−$11.8k−$13.9k
5$8.9k−$2.5k−$3.3k−$14.1k
6−$5.1k−$11.1k−$5.9k−$12.2k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$20.5k$15k$14.1k
2−$598$1.8k$2.3k$3k
3$5.1k$3.9k$4.5k$4.4k
4$9.1k$5.7k$5.8k$7.5k
5$23.5k$23.4k$26k$19.7k
6$4.6k$2.2k$5.3k$4.1k

DeepSeekDeepSeek V4.1 Flash

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$92−$4.1k−$439
2−$2.6k−$5k−$4.3k−$3.6k
3$616−$6.5k−$6.2k−$6.1k
4$8.1k−$6k−$2.7k−$1.5k
5$18.3k−$14.8k−$14.5k−$13.2k
6$557−$5.8k−$6k−$8.2k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$9.7k−$2.5k−$9.8k
2−$6.1k−$8.9k−$8.1k−$7.8k
3−$3.7k−$7.1k−$9.4k−$8.9k
4$5.7k−$5.2k−$8.9k−$11.3k
5$8.9k−$22k−$23.9k−$13.1k
6−$5.1k−$12.3k−$11.6k−$13.9k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$13.6k$15.9k$15.9k
2−$598$1.5k$1.9k$2.2k
3$5.1k$4.1k$4.6k$3.7k
4$9.1k$6.8k$6.6k$6.4k
5$23.5k$15.9k$14.3k$19.8k
6$4.6k$1.6k$2.1k$2.7k

xAIGrok 4.7

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k$209$890−$5.7k
2−$2.6k−$5.1k−$5k−$6.4k
3$616−$9k−$8.4k−$6.4k
4$8.1k−$4.5k−$6.6k−$2.5k
5$18.3k−$18.8k−$18.7k−$20.6k
6$557−$7.8k−$10k−$8.7k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$9.7k−$9.3k−$7.7k
2−$6.1k−$10.4k−$8.3k−$10.3k
3−$3.7k−$8.7k−$9.5k−$9.1k
4$5.7k−$9k−$5.9k−$13.8k
5$8.9k−$21.7k−$19.5k−$19.4k
6−$5.1k−$12.9k−$14.6k−$14.7k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$11.3k$12.3k$14.1k
2−$598$1.1k$1.4k$1.6k
3$5.1k$4.4k$3k$3.1k
4$9.1k$4k$4k$5.5k
5$23.5k$10.1k$8.3k$12.9k
6$4.6k$3k$1k$1.6k

MetaMuse Ember

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$12.5k−$3.1k−$10.6k
2−$2.6k−$5.6k−$6.4k−$1.7k
3$616−$6.6k−$2.8k−$7.1k
4$8.1k−$5.3k−$2k−$4.1k
5$18.3k−$22.5k−$5.8k−$17.5k
6$557−$3.6k−$3.1k−$9.7k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$21.6k−$26.1k−$3.3k
2−$6.1k−$9.9k−$8.1k−$9k
3−$3.7k−$6.4k−$2.6k−$7.5k
4$5.7k−$17.5k−$16.1k−$5.7k
5$8.9k−$23.1k−$16k−$29.5k
6−$5.1k−$12.3k−$11.7k−$8.4k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$17.6k$3.2k$13.8k
2−$598$1.5k$1.2k$425
3$5.1k$4k−$2.8k$1.7k
4$9.1k−$2.2k$3.9k$6.6k
5$23.5k$2.5k$14k$6.1k
6$4.6k$2k$1.2k−$762

MetaMuse Spark 1.3

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$1.2k−$10k−$16.3k
2−$2.6k−$4.7k−$4.6k−$4.1k
3$616−$3.2k−$3.5k−$3.9k
4$8.1k−$5.8k−$4.6k−$5.9k
5$18.3k−$18.5k−$19.7k−$15.1k
6$557−$6.1k−$7.8k−$7.7k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$12.1k−$14.4k−$17.6k
2−$6.1k−$10.2k−$9.8k−$10.5k
3−$3.7k−$11.3k−$5.7k−$9.7k
4$5.7k−$17.4k−$13.1k−$11.7k
5$8.9k−$24.4k−$23.8k−$25.7k
6−$5.1k−$11.5k−$4.1k−$12.5k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$10.2k$12.3k$15.4k
2−$598$1.3k$1.4k$26
3$5.1k$2.8k$1.8k−$136
4$9.1k−$5.1k$5.8k$3.5k
5$23.5k$5.4k$7.6k$10.9k
6$4.6k$3k$3.3k$2.2k

Z.aiGLM 5.3

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$3.7k−$4.7k$1.1k
2−$2.6k−$8.5k−$1.5k−$6.1k
3$616−$4.7k−$8.1k−$6.9k
4$8.1k−$6.4k−$8.6k−$11k
5$18.3k−$21.8k−$21.9k−$23.9k
6$557−$11.6k−$6.1k−$7.3k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$11.6k−$16k−$15.5k
2−$6.1k−$10.6k−$10.4k−$7.7k
3−$3.7k−$7.6k−$6.5k−$9.5k
4$5.7k−$11.7k−$11.6k−$5k
5$8.9k−$15.3k−$24.6k−$28.6k
6−$5.1k−$10.3k–−$13k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$4.9k$9.4k$9.6k
2−$598−$269−$297−$1.7k
3$5.1k$1.8k$217−$9.9k
4$9.1k$1.6k−$602−$911
5$23.5k$11.2k$13.8k$4.8k
6$4.6k−$1.1k−$3.2k$1.2k

AnthropicClaude Haiku 5.5

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$20.2k−$6.7k−$6.6k
2−$2.6k−$10.3k−$4.2k−$3.9k
3$616−$12.8k−$5.5k−$5.7k
4$8.1k−$6.5k−$3.8k−$6.9k
5$18.3k$7.2k−$34.2k−$9.1k
6$557−$7.7k−$10.7k−$20.9k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$20.6k−$17k−$2k
2−$6.1k−$9.1k−$8.5k−$7.5k
3−$3.7k−$9.2k−$6.3k−$11.6k
4$5.7k−$21.4k−$16.6k−$1.5k
5$8.9k−$29.1k−$27.3k−$22.7k
6−$5.1k−$11k−$12.2k−$8.5k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k−$7.5k$11k$3.9k
2−$598−$3.4k$1.5k$1.8k
3$5.1k−$8.1k$1.5k$346
4$9.1k$1.4k$5.3k$7.1k
5$23.5k$5.4k−$1.9k−$10.4k
6$4.6k−$3.1k$1.7k$2.2k

MetaMuse Wonder 1.0

Customer quarter
BusinessFixed ruleRun 1Run 2Run 3
1$16.9k−$16.1k−$15.2k−$14.1k
2−$2.6k−$7.7k−$4.5k−$6.4k
3$616−$5.6k−$6.4k−$6.2k
4$8.1k−$6.6k−$11.2k−$7.8k
5$18.3k−$20.8k−$28.5k−$22.3k
6$557−$8.3k−$7.1k−$12.9k
Supplier quarter
BusinessFixed ruleRun 1Run 2Run 3
1$13.7k−$23k−$20.4k−$23.2k
2−$6.1k−$9.9k−$9.4k−$8.9k
3−$3.7k−$5k−$8.6k−$10.1k
4$5.7k−$18.7k−$20.2k−$17k
5$8.9k−$34.7k−$9.4k−$32.2k
6−$5.1k−$13k−$11.9k−$13.2k
Normal quarter
BusinessFixed ruleRun 1Run 2Run 3
1$18.6k$2.9k$3.8k$6.6k
2−$598$84$151$994
3$5.1k$3.2k$610$2.7k
4$9.1k$3.2k$1.5k$1.4k
5$23.5k$10.9k$4.3k$857
6$4.6k$5.3k−$910$895

OpenAIGPT-6 Luna