Which AI protects a small business’s money best?
We score models on a messy bakery’s problems in a private evaluation, then index
that score with the public benchmarks closest to real owner work.
Updated : Claude Opus 5.5, GPT-6 Astra and GPT-6.1 Sol are tied at the top, protecting $85,989 to $87,578 of $100,000 on average across their runs, with overlapping run ranges. Claude Opus 5.5 has the highest Main Street Index, 63 at $3.49 a task.
Ranking // Dollars Protected Index · $ per task
- [ 1 ]Claude Opus 5.5 Anthropic$87,578 of $100,000 protected63$3.49
- [ 2 ]GPT-6 Astra OpenAI$86,450 of $100,000 protected61$3.86
- [ 3 ]GPT-6.1 Sol OpenAI$85,989 of $100,000 protected60$0.69
- [ 4 ]Kimi K3.1 Pro Moonshot AI$78,150 of $100,000 protected57$2.60
- [ 5 ]Gemini 4 Argon Google$74,917 of $100,000 protected53$3.91
- [ 6 ]Grok 4.7 xAI$74,267 of $100,000 protected50$3.40
- [ 7 ]Claude Sonnet 5.5 Anthropic$73,217 of $100,000 protected50$1.98
- [ 8 ]DeepSeek V4.1 Flash 1017 DeepSeek$72,717 of $100,000 protected48$0.24
- [ 9 ]Muse Wonder Meta$69,125 of $100,000 protected49$3.19
- [ 10 ]GPT-6.1 Terra OpenAI$64,800 of $100,000 protected40$0.19
- [ 11 ]GLM 5.3 Z.ai$62,833 of $100,000 protected38$2.11
- [ 12 ]DeepSeek V4.1 Flash DeepSeek$59,642 of $100,000 protected34$0.30
- [ 13 ]Kimi K3 Moonshot AI$59,067 of $100,000 protected36$2.74
- [ 14 ]Gemini 3.8 Flash Google$57,267 of $100,000 protected32$2.17
- [ 15 ]Muse Spark Open Meta$50,336 of $100,000 protected28$2.84
Leaderboard
| # | Model | v0.14 | v1.1 | v1.0.6 | v2.2 | |||
|---|---|---|---|---|---|---|---|---|
| 01 | Claude Opus 5.5Anthropic | 63 (best)$3.49 | $87,57816 min | 75 (best)64–84$3.94 | 1688 (best)$7.65 | 63.1$0.74 | 27.0$0.86 | 70.2$28.12 |
| 02 | GPT-6 AstraOpenAI | 61$3.86 | $86,45016 min | 7355–83$5.13 | 1503$5.09 | 66.1$1.30 | 31.2$1.81 | 65.0$21.28 |
| 03 | GPT-6.1 SolOpenAI | 60$0.69 | $85,98920 min | 7269–78$0.96 | 1443$0.78 | 64.5$0.23 | 31.3$0.35 | 63.9$3.61 |
| 04 | Kimi K3.1 ProMoonshot AI | 57$2.60 | $78,15013 min | 5751–61$2.30 | 1676$7.22 | 73.3$0.83 | 31.4 (best)$0.52 | 62.5$6.44 |
| 05 | Gemini 4 ArgonGoogle | 53$3.91 | $74,91716 min | 5028–58$4.25 | 1484$9.17 | 77.5 (best)$1.12 | 21.8$0.52 | 69.2$13.37 |
| 06 | Grok 4.7xAI | 50$3.40 | $74,26726 min | 4946–50$4.39 | 1629$6.38 | 63.5$0.85 | 23.2$0.30 | 58.6$11.63 |
| 07 | Claude Sonnet 5.5Anthropic | 50$1.98 | $73,21712 min | 4722–66$2.48 | 1606$3.88 | 60.0$0.33 | 23.3$0.42 | 70.8 (best)$18.70 |
| 08 | DeepSeek V4.1 Flash 1017DeepSeek | 48$0.24 | $72,71714 min | 4527–71$0.18 | 1570$0.68 | 66.6$0.17 | 17.2$0.040 | 61.3$0.66 |
| 09 | Muse WonderMeta | 49$3.19 | $69,12512 min | 390–49$2.98 | 1659$8.68 | 68.1$0.72 | 30.6$0.72 | 64.9$10.65 |
| 10 | GPT-6.1 TerraOpenAI | 40$0.19 | $64,8003 min | 3020–38$0.18 | 1263$0.29 | 58.7$0.11 | 27.7$0.17 | 60.5$2.94 |
| 11 | GLM 5.3Z.ai | 38$2.11 | $62,83322 min | 2616–40$2.13 | 1505$5.65 | 62.2$0.46 | 11.2$0.17 | 59.7$8.52 |
| 12 | DeepSeek V4.1 FlashDeepSeek | 34$0.30 | $59,6429 min | 196–34$0.25 | 1417$0.57 | 68.9$0.17 | 12.8— | 51.3$0.33 |
| 13 | Kimi K3Moonshot AI | 36$2.74 | $59,06719 min | 220–34$2.04 | 1498$6.73 | 58.3$0.63 | 22.0— | 53.0$7.17 |
| 14 | Gemini 3.8 FlashGoogle | 32$2.17 | $57,26712 min | 158–25$1.41 | 1203$7.52 | 59.9$0.90 | 21.0$0.11 | 56.0$5.03 |
| 15 | Muse Spark OpenMeta | 28$2.84 | $50,33624 min | 120–43$5.25 | 1279$1.47 | 49.7$0.46 | 18.6$0.19 | 48.6$4.72 |
Main Street Bench runs are scored on version 0.14, and run to run shows them all. Highlighted: the best in its column. Bench figures are the mean of each model’s runs at the effort levels people most often choose, because providers’ effort labels aren’t like for like on thinking budget, and the range under a score is its lowest and highest run. Artificial Analysis figures are blended the same way, or are a model’s one listed entry. Artificial Analysis doesn’t list a GDP.pdf cost for Kimi K3 or DeepSeek V4.1 Flash, so their index cost leaves it out. Vals scores are from version 2.2 of the Vals Index, except DeepSeek V4.1 Flash’s, which it doesn’t list; those are from version 2.1, which scores most models a few points lower.
Cost Frontier
Most attractive quadrantPareto line
Pareto optimal Index
- Claude Opus 5.563$3.49 / task
- GPT-6.1 Sol60$0.69 / task
- DeepSeek V4.1 Flash 101748$0.24 / task
- GPT-6.1 Terra40$0.19 / task
Run to run
Main Street Bench score
Dollars protected
The benchmarks
What goes into the index
- 40%100problems
Main Street Bench
One bakery's website, books, inbox, menu and policies, with 100 hidden problems that put $100,000 at risk. Score is the money a model protects beyond the basics.
Ours · kept private - 15%91tasks
AA-Briefcase
Multi-week office projects built from piles of real files: emails, chat threads, meeting notes, spreadsheets. Graded on the reports the model hands back.
Artificial Analysis · Elo - 15%657tasks
AutomationBench-AA
Multi-step admin in finance, HR, marketing, operations, sales and support, done inside Gmail, Sheets, Slack, Salesforce and Zendesk.
Artificial Analysis · % of steps done - 15%100tasks
GDP.pdf
Questions on long professional PDFs in fields such as HR, law, insurance, finance, real estate and construction, passed only if every criterion is met.
Artificial Analysis · % all-pass - 15%8benchmarks
Vals Index
Agentic work in finance, coding, law and tax, from financial models in spreadsheets to cited tax research, weighted by each sector's share of the U.S. economy.
Vals AI · % accuracy
Our Methodology
One message from the owner and one folder with everything for the business. 100 problems are hidden in it, with $100,000 at risk between them.
“Hi! I own Main Street Bakery, a small neighborhood bakery.
Something feels off with the business lately, and I can’t put my finger on it. I’ve attached a folder with everything for the business: our ordering website, September’s books, the menu and allergen sheet, our policies, our online listing and the shop’s emails.
Could you go through all of it? Please fix whatever you can, but don’t break anything that’s working. Customers have real orders on the website, so please don’t lose any of them. If something needs me to sort it out, like money we’re owed, tell me exactly what’s wrong and which order, payment or invoice numbers to look at.
When you’re done, explain it to me like I’m not a tech person: what you found, what you fixed, and what I still need to do.”
- Emails
- 42
- Live website orders
- 43
- Spreadsheets
- 17
- Spreadsheet rows
- 2,390
admin/9Accounts and services, the lease, September’s payroll, the rota, staff, promotions and newsletter subscribersbooks/13September’s register exports, card payments, payouts, refunds, the bank statement, cash counts, supplier invoices and the draft sales tax returndrafts/4Replies and a newsletter written but not sentinbox/42Emails from the last few weeks, one per filelisting/2The online business listing and its reviewsmenu/7The menu, price lists, the board in the shop, the allergen sheet, recipes and the coffee supplier’s specpolicies/5Refunds, cancellations, gift cards, coupons and the receipt templatesigns/1The sign on the front doorwebsite/22The online ordering site’s code and tests, and the database with customers’ live orders
Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5Grok 4.7Gemini 3.8 FlashGPT-6 AstraMuse WonderGemini 4 ArgonKimi K3GLM 5.3DeepSeek V4.1 FlashMuse Spark OpenDeepSeek V4.1 Flash 1017Kimi K3.1 ProGPT-6.1 Terra
- Money and books27222322202119202022181919121110192523231919321232561615141517131417130710171620241918121514
- Website and ordering251920201218141110101112127681912201616141116181510111088978810101014101111151414111110
- Customers and inbox2222202219202019171812121291181719181088312121279101416787814034191722171916788
- Policies and legal17141010141316139131412156910151515131210614101481278785455171112911121111789
- Menu, prices and allergens866588866677856678877677773675564544566677777665
- Online listing111111110111100011111111000000000011001111111101
- 2Every model, every run
- 93Fixed in some runs, not others
- 5No model, any run
How the score works
The score is in dollars. Each problem carries what it could cost the shop within a year, and the 100 add up to $100,000. The score is the share of what a model protects, less what it loses to problems missed, things broken and fixes it wrongly claimed.
Scorecards
107 runs · fifteen models Show the scorecardsHide them
Medium effort · 32 runs · each with its scorecard
Claude Opus 5.5
75 out of 100Mean of three medium runs
81 of 100 fixed · $87,683 of $100,000 protected, on average
GPT-6.1 Sol
72 out of 100Mean of three medium runs
78 of 100 fixed · $86,033 of $100,000 protected, on average
GPT-6 Astra
71 out of 100Mean of three medium runs
81 of 100 fixed · $85,600 of $100,000 protected, on average
Gemini 4 Argon
55 out of 100Mean of three medium runs
71 of 100 fixed · $77,300 of $100,000 protected, on average
Claude Sonnet 5.5
49 out of 100Mean of three medium runs
67 of 100 fixed · $74,600 of $100,000 protected, on average
Grok 4.7
48 out of 100Mean of three medium runs
64 of 100 fixed · $74,050 of $100,000 protected, on average
Muse Wonder
32 out of 100Mean of four medium runs
56 of 100 fixed · $65,062 of $100,000 protected, on average
- Run 149 out of 10070 fixed $74,6507 min ~$3.45Scorecard
- Run 245 out of 10063 fixed $72,6506 min ~$2.76Scorecard
- Run 335 out of 10058 fixed $67,45030 min ~$2.34Scorecard
- Run 40 out of 10031 fixed $45,50038 min ~$2.07Scorecard
This run hung mid-work and never reported back to the owner, so it counts as a 0.
GPT-6.1 Terra
24 out of 100Mean of three medium runs
46 of 100 fixed · $61,967 of $100,000 protected, on average
Gemini 3.8 Flash
17 out of 100Mean of three medium runs
41 of 100 fixed · $58,383 of $100,000 protected, on average
Muse Spark Open
4 out of 100Mean of four medium runs
36 of 100 fixed · $42,425 of $100,000 protected, on average
- Run 117 out of 10047 fixed $58,60017 min $2.47Scorecard
- Run 20 out of 10016 fixed $25,00030 min $2.78Scorecard
This run’s helper agents kept working until the 30-minute limit and its report was one opening line, so it counts as a 0.
- Run 30 out of 10033 fixed $38,30030 min $8.25Scorecard
- Run 40 out of 10046 fixed $47,80028 min $3.54Scorecard
Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro weren’t run at medium.
High effort · 42 runs
GPT-6.1 Sol
74 out of 100Mean of three high runs
$86,917 of $100,000 protected, on average
- Run 171 out of 100$85,35012 min ~$0.71
- Run 273 out of 100$86,45022 min ~$1.15
- Run 378 out of 100$88,95019 min ~$0.99
GPT-6 Astra
74 out of 100Mean of three high runs
$86,650 of $100,000 protected, on average
- Run 175 out of 100$87,25017 min ~$5.19
- Run 275 out of 100$87,45016 min ~$5.25
- Run 371 out of 100$85,25017 min ~$5.08
Claude Opus 5.5
70 out of 100Mean of three high runs
$85,083 of $100,000 protected, on average
- Run 176 out of 100$88,10012 min $2.74
- Run 264 out of 100$81,90010 min $2.26
- Run 371 out of 100$85,2508 min $2.37
Kimi K3.1 Pro
57 out of 100Mean of three high runs
71 of 100 fixed · $78,150 of $100,000 protected, on average
Claude Sonnet 5.5
55 out of 100Mean of three high runs
$77,317 of $100,000 protected, on average
- Run 162 out of 100$80,90011 min $2.45
- Run 236 out of 100$68,15011 min $2.14
- Run 366 out of 100$82,90010 min $2.17
DeepSeek V4.1 Flash 1017
49 out of 100Mean of three high runs
66 of 100 fixed · $74,667 of $100,000 protected, on average
Grok 4.7
49 out of 100Mean of three high runs
$74,267 of $100,000 protected, on average
- Run 146 out of 100$73,05023 min ~$4.36
- Run 250 out of 100$74,80027 min —
- Run 350 out of 100$74,95026 min ~$4.43
Gemini 4 Argon
45 out of 100Mean of three high runs
$72,533 of $100,000 protected, on average
- Run 128 out of 100$63,75020 min ~$5.18
- Run 257 out of 100$78,65015 min ~$4.39
- Run 350 out of 100$75,20019 min ~$5.37
Muse Wonder
44 out of 100Mean of three high runs
$71,933 of $100,000 protected, on average
- Run 142 out of 100$71,0505 min ~$2.83
- Run 247 out of 100$73,3006 min ~$2.53
- Run 343 out of 100$71,4506 min ~$3.45
GPT-6.1 Terra
31 out of 100Mean of three high runs
$65,550 of $100,000 protected, on average
- Run 134 out of 100$67,0504 min ~$0.19
- Run 232 out of 100$66,0503 min ~$0.20
- Run 327 out of 100$63,5502 min ~$0.16
GLM 5.3
25 out of 100Mean of three high runs
49 of 100 fixed · $62,250 of $100,000 protected, on average
Gemini 3.8 Flash
15 out of 100Mean of three high runs
$57,267 of $100,000 protected, on average
- Run 125 out of 100$62,30012 min ~$1.51
- Run 28 out of 100$54,20012 min ~$1.30
- Run 311 out of 100$55,30012 min ~$1.41
Kimi K3
22 out of 100Mean of three high runs
46 of 100 fixed · $56,700 of $100,000 protected, on average
DeepSeek V4.1 Flash
10 out of 100Mean of three high runs
39 of 100 fixed · $55,267 of $100,000 protected, on average
Muse Spark Open wasn’t run at high.
Xhigh effort · eighteen runs
Claude Opus 5.5
80 out of 100Mean of three xhigh runs
$89,967 of $100,000 protected, on average
- Run 182 out of 100$90,85029 min $6.86
- Run 274 out of 100$86,95026 min $6.54
- Run 384 out of 100$92,10029 min $7.10
GPT-6 Astra
74 out of 100Mean of three xhigh runs
$87,100 of $100,000 protected, on average
- Run 183 out of 100$91,70020 min ~$6.14
- Run 260 out of 100$79,90020 min ~$6.39
- Run 379 out of 100$89,70019 min ~$7.25
GPT-6.1 Sol
70 out of 100Mean of three xhigh runs
$85,017 of $100,000 protected, on average
- Run 171 out of 100$85,60028 min ~$1.27
- Run 269 out of 100$84,45027 min ~$1.24
- Run 370 out of 100$85,00027 min ~$1.17
Muse Wonder
43 out of 100Mean of three xhigh runs
$71,733 of $100,000 protected, on average
- Run 144 out of 100$71,8006 min ~$3.04
- Run 239 out of 100$69,7006 min ~$3.11
- Run 347 out of 100$73,7008 min ~$4.21
Claude Sonnet 5.5
36 out of 100Mean of three xhigh runs
$67,733 of $100,000 protected, on average
- Run 131 out of 100$65,40016 min $3.43
- Run 235 out of 100$67,40015 min $3.40
- Run 341 out of 100$70,40017 min $3.96
GPT-6.1 Terra
34 out of 100Mean of three xhigh runs
$66,883 of $100,000 protected, on average
- Run 135 out of 100$67,5503 min ~$0.20
- Run 229 out of 100$64,3004 min ~$0.21
- Run 338 out of 100$68,8005 min ~$0.26
Grok 4.7, Gemini 3.8 Flash, Gemini 4 Argon, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, Muse Spark Open, DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro weren’t run at xhigh.
Max effort · fifteen runs
DeepSeek V4.1 Flash 1017
41 out of 100Mean of three max runs
$70,767 of $100,000 protected, on average
- Run 137 out of 100$68,65012 min $0.29
- Run 251 out of 100$75,65014 min $0.14
- Run 336 out of 100$68,00011 min $0.11
DeepSeek V4.1 Flash
28 out of 100Mean of three max runs
$64,017 of $100,000 protected, on average
- Run 134 out of 100$66,9509 min ~$0.26
- Run 223 out of 100$61,55019 min ~$0.34
- Run 327 out of 100$63,55010 min ~$0.23
GLM 5.3
27 out of 100Mean of three max runs
$63,417 of $100,000 protected, on average
- Run 116 out of 100$57,80025 min ~$2.08
- Run 225 out of 100$62,30027 min ~$3.18
- Run 340 out of 100$70,15036 min ~$2.12
Kimi K3
23 out of 100Mean of three max runs
$61,433 of $100,000 protected, on average
- Run 131 out of 100$65,7006 min ~$2.90
- Run 23 out of 100$51,55030 min ~$1.92
- Run 334 out of 100$67,05030 min ~$1.88
Muse Spark Open
23 out of 100Mean of three max runs
$60,883 of $100,000 protected, on average
- Run 10 out of 100$49,05022 min $4.15
- Run 225 out of 100$62,30028 min $3.58
- Run 343 out of 100$71,30012 min $11.96
Claude Opus 5.5, GPT-6.1 Sol, Claude Sonnet 5.5, Grok 4.7, Gemini 3.8 Flash, GPT-6 Astra, Muse Wonder, Gemini 4 Argon, Kimi K3.1 Pro and GPT-6.1 Terra weren’t run at max.
Each card is one run: its score, problems fixed (the runs behind the ticks only), dollars protected, time and cost (~ worked out from tokens, — where the tool’s count fell short). Each scorecard is the benchmark’s own and gives counts only.