Main Street Bench

Which AI protects a small business’s money best?
We score models on a messy bakery’s problems in a private evaluation, then index that score with the public benchmarks closest to real owner work.

Updated : Claude Opus 5.5, GPT-6 Astra and GPT-6.1 Sol are tied at the top, protecting $85,989 to $87,578 of $100,000 on average across their runs, with overlapping run ranges. Claude Opus 5.5 has the highest Main Street Index, 63 at $3.49 a task.

Ranking // Dollars Protected Index · $ per task

  1. [ 1 ]Claude Opus 5.5 Anthropic$87,578 of $100,000 protected63$3.49
  2. [ 2 ]GPT-6 Astra OpenAI$86,450 of $100,000 protected61$3.86
  3. [ 3 ]GPT-6.1 Sol OpenAI$85,989 of $100,000 protected60$0.69
  4. [ 4 ]Kimi K3.1 Pro Moonshot AI$78,150 of $100,000 protected57$2.60
  5. [ 5 ]Gemini 4 Argon Google$74,917 of $100,000 protected53$3.91
  6. [ 6 ]Grok 4.7 xAI$74,267 of $100,000 protected50$3.40
  7. [ 7 ]Claude Sonnet 5.5 Anthropic$73,217 of $100,000 protected50$1.98
  8. [ 8 ]DeepSeek V4.1 Flash 1017 DeepSeek$72,717 of $100,000 protected48$0.24
  9. [ 9 ]Muse Wonder Meta$69,125 of $100,000 protected49$3.19
  10. [ 10 ]GPT-6.1 Terra OpenAI$64,800 of $100,000 protected40$0.19
  11. [ 11 ]GLM 5.3 Z.ai$62,833 of $100,000 protected38$2.11
  12. [ 12 ]DeepSeek V4.1 Flash DeepSeek$59,642 of $100,000 protected34$0.30
  13. [ 13 ]Kimi K3 Moonshot AI$59,067 of $100,000 protected36$2.74
  14. [ 14 ]Gemini 3.8 Flash Google$57,267 of $100,000 protected32$2.17
  15. [ 15 ]Muse Spark Open Meta$50,336 of $100,000 protected28$2.84

Leaderboard

#Modelv0.14v1.1v1.0.6v2.2
01Claude Opus 5.5Anthropic63 (best)$3.49$87,57816 min75 (best)64–84$3.941688 (best)$7.6563.1$0.7427.0$0.8670.2$28.12
02GPT-6 AstraOpenAI61$3.86$86,45016 min7355–83$5.131503$5.0966.1$1.3031.2$1.8165.0$21.28
03GPT-6.1 SolOpenAI60$0.69$85,98920 min7269–78$0.961443$0.7864.5$0.2331.3$0.3563.9$3.61
04Kimi K3.1 ProMoonshot AI57$2.60$78,15013 min5751–61$2.301676$7.2273.3$0.8331.4 (best)$0.5262.5$6.44
05Gemini 4 ArgonGoogle53$3.91$74,91716 min5028–58$4.251484$9.1777.5 (best)$1.1221.8$0.5269.2$13.37
06Grok 4.7xAI50$3.40$74,26726 min4946–50$4.391629$6.3863.5$0.8523.2$0.3058.6$11.63
07Claude Sonnet 5.5Anthropic50$1.98$73,21712 min4722–66$2.481606$3.8860.0$0.3323.3$0.4270.8 (best)$18.70
08DeepSeek V4.1 Flash 1017DeepSeek48$0.24$72,71714 min4527–71$0.181570$0.6866.6$0.1717.2$0.04061.3$0.66
09Muse WonderMeta49$3.19$69,12512 min390–49$2.981659$8.6868.1$0.7230.6$0.7264.9$10.65
10GPT-6.1 TerraOpenAI40$0.19$64,8003 min3020–38$0.181263$0.2958.7$0.1127.7$0.1760.5$2.94
11GLM 5.3Z.ai38$2.11$62,83322 min2616–40$2.131505$5.6562.2$0.4611.2$0.1759.7$8.52
12DeepSeek V4.1 FlashDeepSeek34$0.30$59,6429 min196–34$0.251417$0.5768.9$0.1712.8—51.3$0.33
13Kimi K3Moonshot AI36$2.74$59,06719 min220–34$2.041498$6.7358.3$0.6322.0—53.0$7.17
14Gemini 3.8 FlashGoogle32$2.17$57,26712 min158–25$1.411203$7.5259.9$0.9021.0$0.1156.0$5.03
15Muse Spark OpenMeta28$2.84$50,33624 min120–43$5.251279$1.4749.7$0.4618.6$0.1948.6$4.72
Main Street Index Our overall rating, out of 100: the Main Street Bench score counts 40% and the four public benchmarks 15% each, each put on a 0–100 scale first. Under it: the cost of a task, weighted the same way. Main Street Bench Of the $100,000 that 100 hidden problems put at risk, what a model kept safe, less what it broke or wrongly said it fixed. The mean of its runs at the effort levels people most often choose. Under it: the time of one full run. AA-Briefcase An Elo rating of the reports and spreadsheets a model hands back. Under it: Artificial Analysis's cost per task. AutomationBench-AA The percentage of task steps done correctly, on version 1.0.6 of the dataset. GDP.pdf The percentage of attempts that meet every criterion of a question about a long professional PDF. Under it: Artificial Analysis's cost per task, where it lists one. Vals Index Accuracy on eight agentic benchmarks in finance, coding, legal and tax, at each model’s default effort, which Vals AI ranks on and which isn’t always its highest. Under it: its cost per test, which the index cost leaves out. Main Street Bench score Main Street Bench alone, out of 100: the share of the $50,000 beyond the 35 basics a model protected, less what it lost. The mean of its runs at the effort levels people most often choose, and the part the index uses. Then its lowest and highest run. Under it: the cost of one full run, worked out from tokens for every model but Opus, Sonnet and Muse Spark Open.

Main Street Bench runs are scored on version 0.14, and run to run shows them all. Highlighted: the best in its column. Bench figures are the mean of each model’s runs at the effort levels people most often choose, because providers’ effort labels aren’t like for like on thinking budget, and the range under a score is its lowest and highest run. Artificial Analysis figures are blended the same way, or are a model’s one listed entry. Artificial Analysis doesn’t list a GDP.pdf cost for Kimi K3 or DeepSeek V4.1 Flash, so their index cost leaves it out. Vals scores are from version 2.2 of the Vals Index, except DeepSeek V4.1 Flash’s, which it doesn’t list; those are from version 2.1, which scores most models a few points lower.

Cost Frontier

Most attractive quadrantPareto line

Claude Opus 5.5: index 63 at $3.49 per task. GPT-6 Astra: index 61 at $3.86 per task. GPT-6.1 Sol: index 60 at $0.69 per task. Kimi K3.1 Pro: index 57 at $2.60 per task. Gemini 4 Argon: index 53 at $3.91 per task. Claude Sonnet 5.5: index 50 at $1.98 per task. Grok 4.7: index 50 at $3.40 per task. Muse Wonder: index 49 at $3.19 per task. DeepSeek V4.1 Flash 1017: index 48 at $0.24 per task. GPT-6.1 Terra: index 40 at $0.19 per task. GLM 5.3: index 38 at $2.11 per task. Kimi K3: index 36 at $2.74 per task. DeepSeek V4.1 Flash: index 34 at $0.30 per task. Gemini 3.8 Flash: index 32 at $2.17 per task. Muse Spark Open: index 28 at $2.84 per task. 20 30 40 50 60 70 $0.1 $0.3 $1 $3 $10 Cost per task (USD, log scale) Main Street Index Claude Opus 5.5: index 63, $3.49 per task GPT-6 Astra: index 61, $3.86 per task GPT-6.1 Sol: index 60, $0.69 per task Kimi K3.1 Pro: index 57, $2.60 per task Gemini 4 Argon: index 53, $3.91 per task Claude Sonnet 5.5: index 50, $1.98 per task Grok 4.7: index 50, $3.40 per task Muse Wonder: index 49, $3.19 per task DeepSeek V4.1 Flash 1017: index 48, $0.24 per task GPT-6.1 Terra: index 40, $0.19 per task GLM 5.3: index 38, $2.11 per task Kimi K3: index 36, $2.74 per task DeepSeek V4.1 Flash: index 34, $0.30 per task Gemini 3.8 Flash: index 32, $2.17 per task Muse Spark Open: index 28, $2.84 per task

Pareto optimal Index

  1. Claude Opus 5.563$3.49 / task
  2. GPT-6.1 Sol60$0.69 / task
  3. DeepSeek V4.1 Flash 101748$0.24 / task
  4. GPT-6.1 Terra40$0.19 / task

Run to run

Main Street Bench score

Claude Opus 5.5: mean 75, runs from 64 to 84. GPT-6 Astra: mean 73, runs from 55 to 83. GPT-6.1 Sol: mean 72, runs from 69 to 78. Kimi K3.1 Pro: mean 57, runs from 51 to 61. Gemini 4 Argon: mean 50, runs from 28 to 58. Grok 4.7: mean 49, runs from 46 to 50. Claude Sonnet 5.5: mean 47, runs from 22 to 66. DeepSeek V4.1 Flash 1017: mean 45, runs from 27 to 71. Muse Wonder: mean 39, runs from 0 to 49. GPT-6.1 Terra: mean 30, runs from 20 to 38. GLM 5.3: mean 26, runs from 16 to 40. Kimi K3: mean 22, runs from 0 to 34. DeepSeek V4.1 Flash: mean 19, runs from 6 to 34. Gemini 3.8 Flash: mean 15, runs from 8 to 25. Muse Spark Open: mean 12, runs from 0 to 43. 0 10 20 30 40 50 60 70 80 90 100 SCORE OUT OF 100 Claude Opus 5.5: mean 75 of 9 runs, lowest 64, highest 84CLAUDE OPUS 5.575 · 64–84 GPT-6 Astra: mean 73 of 9 runs, lowest 55, highest 83GPT-6 ASTRA73 · 55–83 GPT-6.1 Sol: mean 72 of 9 runs, lowest 69, highest 78GPT-6.1 SOL72 · 69–78 Kimi K3.1 Pro: mean 57 of 3 runs, lowest 51, highest 61KIMI K3.1 PRO57 · 51–61 Gemini 4 Argon: mean 50 of 6 runs, lowest 28, highest 58GEMINI 4 ARGON50 · 28–58 Grok 4.7: mean 49 of 3 runs, lowest 46, highest 50GROK 4.749 · 46–50 Claude Sonnet 5.5: mean 47 of 9 runs, lowest 22, highest 66CLAUDE SONNET 5.547 · 22–66 DeepSeek V4.1 Flash 1017: mean 45 of 6 runs, lowest 27, highest 71DEEPSEEK V4.1 FLASH 101745 · 27–71 Muse Wonder: mean 39 of 10 runs, lowest 0, highest 49MUSE WONDER39 · 0–49 GPT-6.1 Terra: mean 30 of 9 runs, lowest 20, highest 38GPT-6.1 TERRA30 · 20–38 GLM 5.3: mean 26 of 6 runs, lowest 16, highest 40GLM 5.326 · 16–40 Kimi K3: mean 22 of 6 runs, lowest 0, highest 34KIMI K322 · 0–34 DeepSeek V4.1 Flash: mean 19 of 6 runs, lowest 6, highest 34DEEPSEEK V4.1 FLASH19 · 6–34 Gemini 3.8 Flash: mean 15 of 3 runs, lowest 8, highest 25GEMINI 3.8 FLASH15 · 8–25 Muse Spark Open: mean 12 of 7 runs, lowest 0, highest 43MUSE SPARK OPEN12 · 0–43

Dollars protected

Claude Opus 5.5: mean $87,578, runs from $81,900 to $92,100. GPT-6 Astra: mean $86,450, runs from $77,450 to $91,700. GPT-6.1 Sol: mean $85,989, runs from $84,450 to $88,950. Kimi K3.1 Pro: mean $78,150, runs from $75,400 to $80,250. Gemini 4 Argon: mean $74,917, runs from $63,750 to $79,050. Grok 4.7: mean $74,267, runs from $73,050 to $74,950. Claude Sonnet 5.5: mean $73,217, runs from $60,750 to $82,900. DeepSeek V4.1 Flash 1017: mean $72,717, runs from $63,500 to $85,500. Muse Wonder: mean $69,125, runs from $45,500 to $74,650. GPT-6.1 Terra: mean $64,800, runs from $60,050 to $68,800. GLM 5.3: mean $62,833, runs from $57,800 to $70,150. DeepSeek V4.1 Flash: mean $59,642, runs from $53,050 to $66,950. Kimi K3: mean $59,067, runs from $37,750 to $67,050. Gemini 3.8 Flash: mean $57,267, runs from $54,200 to $62,300. Muse Spark Open: mean $50,336, runs from $25,000 to $71,300. $20k $30k $40k $50k $60k $70k $80k $90k $100k $ PROTECTED OF $100,000 Claude Opus 5.5: mean $87,578 of 9 runs, lowest $81,900, highest $92,100CLAUDE OPUS 5.5$87,578 · $81,900–$92,100 GPT-6 Astra: mean $86,450 of 9 runs, lowest $77,450, highest $91,700GPT-6 ASTRA$86,450 · $77,450–$91,700 GPT-6.1 Sol: mean $85,989 of 9 runs, lowest $84,450, highest $88,950GPT-6.1 SOL$85,989 · $84,450–$88,950 Kimi K3.1 Pro: mean $78,150 of 3 runs, lowest $75,400, highest $80,250KIMI K3.1 PRO$78,150 · $75,400–$80,250 Gemini 4 Argon: mean $74,917 of 6 runs, lowest $63,750, highest $79,050GEMINI 4 ARGON$74,917 · $63,750–$79,050 Grok 4.7: mean $74,267 of 3 runs, lowest $73,050, highest $74,950GROK 4.7$74,267 · $73,050–$74,950 Claude Sonnet 5.5: mean $73,217 of 9 runs, lowest $60,750, highest $82,900CLAUDE SONNET 5.5$73,217 · $60,750–$82,900 DeepSeek V4.1 Flash 1017: mean $72,717 of 6 runs, lowest $63,500, highest $85,500DEEPSEEK V4.1 FLASH 1017$72,717 · $63,500–$85,500 Muse Wonder: mean $69,125 of 10 runs, lowest $45,500, highest $74,650MUSE WONDER$69,125 · $45,500–$74,650 GPT-6.1 Terra: mean $64,800 of 9 runs, lowest $60,050, highest $68,800GPT-6.1 TERRA$64,800 · $60,050–$68,800 GLM 5.3: mean $62,833 of 6 runs, lowest $57,800, highest $70,150GLM 5.3$62,833 · $57,800–$70,150 DeepSeek V4.1 Flash: mean $59,642 of 6 runs, lowest $53,050, highest $66,950DEEPSEEK V4.1 FLASH$59,642 · $53,050–$66,950 Kimi K3: mean $59,067 of 6 runs, lowest $37,750, highest $67,050KIMI K3$59,067 · $37,750–$67,050 Gemini 3.8 Flash: mean $57,267 of 3 runs, lowest $54,200, highest $62,300GEMINI 3.8 FLASH$57,267 · $54,200–$62,300 Muse Spark Open: mean $50,336 of 7 runs, lowest $25,000, highest $71,300MUSE SPARK OPEN$50,336 · $25,000–$71,300

The benchmarks

What goes into the index

  • 40%100problems

    Main Street Bench

    One bakery's website, books, inbox, menu and policies, with 100 hidden problems that put $100,000 at risk. Score is the money a model protects beyond the basics.

    Ours · kept private
  • 15%91tasks

    AA-Briefcase

    Multi-week office projects built from piles of real files: emails, chat threads, meeting notes, spreadsheets. Graded on the reports the model hands back.

    Artificial Analysis · Elo
  • 15%657tasks

    AutomationBench-AA

    Multi-step admin in finance, HR, marketing, operations, sales and support, done inside Gmail, Sheets, Slack, Salesforce and Zendesk.

    Artificial Analysis · % of steps done
  • 15%100tasks

    GDP.pdf

    Questions on long professional PDFs in fields such as HR, law, insurance, finance, real estate and construction, passed only if every criterion is met.

    Artificial Analysis · % all-pass
  • 15%8benchmarks

    Vals Index

    Agentic work in finance, coding, law and tax, from financial models in spreadsheets to cited tax research, weighted by each sector's share of the U.S. economy.

    Vals AI · % accuracy

Our Methodology

One message from the owner and one folder with everything for the business. 100 problems are hidden in it, with $100,000 at risk between them.

The owner’s AI request

“Hi! I own Main Street Bakery, a small neighborhood bakery.

Something feels off with the business lately, and I can’t put my finger on it. I’ve attached a folder with everything for the business: our ordering website, September’s books, the menu and allergen sheet, our policies, our online listing and the shop’s emails.

Could you go through all of it? Please fix whatever you can, but don’t break anything that’s working. Customers have real orders on the website, so please don’t lose any of them. If something needs me to sort it out, like money we’re owed, tell me exactly what’s wrong and which order, payment or invoice numbers to look at.

When you’re done, explain it to me like I’m not a tech person: what you found, what you fixed, and what I still need to do.”

The attached folder 9 folders · 106 files
Emails
42
Live website orders
43
Spreadsheets
17
Spreadsheet rows
2,390
  • admin/9Accounts and services, the lease, September’s payroll, the rota, staff, promotions and newsletter subscribers
  • books/13September’s register exports, card payments, payouts, refunds, the bank statement, cash counts, supplier invoices and the draft sales tax return
  • drafts/4Replies and a newsletter written but not sent
  • inbox/42Emails from the last few weeks, one per file
  • listing/2The online business listing and its reviews
  • menu/7The menu, price lists, the board in the shop, the allergen sheet, recipes and the coffee supplier’s spec
  • policies/5Refunds, cancellations, gift cards, coupons and the receipt template
  • signs/1The sign on the front door
  • website/22The online ordering site’s code and tests, and the database with customers’ live orders

Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5Grok 4.7Gemini 3.8 FlashGPT-6 AstraMuse WonderGemini 4 ArgonKimi K3GLM 5.3DeepSeek V4.1 FlashMuse Spark OpenDeepSeek V4.1 Flash 1017Kimi K3.1 ProGPT-6.1 Terra

  • Money and books27222322202119202022181919121110192523231919321232561615141517131417130710171620241918121514
  • Website and ordering251920201218141110101112127681912201616141116181510111088978810101014101111151414111110
  • Customers and inbox2222202219202019171812121291181719181088312121279101416787814034191722171916788
  • Policies and legal17141010141316139131412156910151515131210614101481278785455171112911121111789
  • Menu, prices and allergens866588866677856678877677773675564544566677777665
  • Online listing111111110111100011111111000000000011001111111101
  • 2Every model, every run
  • 93Fixed in some runs, not others
  • 5No model, any run
How the score works

The score is in dollars. Each problem carries what it could cost the shop within a year, and the 100 add up to $100,000. The score is the share of what a model protects, less what it loses to problems missed, things broken and fixes it wrongly claimed.

Scorecards

107 runs · fifteen models Show the scorecardsHide them
Which runs

Medium effort · 32 runs · each with its scorecard

  1. Claude Opus 5.5

    75 out of 100Mean of three medium runs

    81 of 100 fixed · $87,683 of $100,000 protected, on average

    1. Run 181 out of 10084 fixed $90,35010 min $2.50Scorecard
    2. Run 274 out of 10080 fixed $87,1008 min $2.31Scorecard
    3. Run 371 out of 10080 fixed $85,60012 min $2.78Scorecard
  2. GPT-6.1 Sol

    72 out of 100Mean of three medium runs

    78 of 100 fixed · $86,033 of $100,000 protected, on average

    1. Run 169 out of 10074 fixed $84,50016 min ~$0.70Scorecard
    2. Run 275 out of 10081 fixed $87,60015 min ~$0.77Scorecard
    3. Run 372 out of 10078 fixed $86,00012 min ~$0.63Scorecard
  3. GPT-6 Astra

    71 out of 100Mean of three medium runs

    81 of 100 fixed · $85,600 of $100,000 protected, on average

    1. Run 155 out of 10078 fixed $77,45012 min ~$3.52Scorecard
    2. Run 280 out of 10080 fixed $89,90013 min ~$3.99Scorecard
    3. Run 379 out of 10085 fixed $89,45012 min ~$3.38Scorecard
  4. Gemini 4 Argon

    55 out of 100Mean of three medium runs

    71 of 100 fixed · $77,300 of $100,000 protected, on average

    1. Run 156 out of 10071 fixed $77,95014 min ~$4.17Scorecard
    2. Run 250 out of 10070 fixed $74,90010 min ~$2.36Scorecard
    3. Run 358 out of 10073 fixed $79,05016 min ~$4.02Scorecard
  5. Claude Sonnet 5.5

    49 out of 100Mean of three medium runs

    67 of 100 fixed · $74,600 of $100,000 protected, on average

    1. Run 164 out of 10070 fixed $82,1508 min $1.52Scorecard
    2. Run 222 out of 10062 fixed $60,7508 min $1.53Scorecard
    3. Run 362 out of 10070 fixed $80,9008 min $1.72Scorecard
  6. Grok 4.7

    48 out of 100Mean of three medium runs

    64 of 100 fixed · $74,050 of $100,000 protected, on average

    1. Run 146 out of 10063 fixed $72,80018 min ~$3.32Scorecard
    2. Run 246 out of 10063 fixed $73,05019 min —Scorecard
    3. Run 353 out of 10067 fixed $76,30020 min ~$3.28Scorecard
  7. Muse Wonder

    32 out of 100Mean of four medium runs

    56 of 100 fixed · $65,062 of $100,000 protected, on average

    1. Run 149 out of 10070 fixed $74,6507 min ~$3.45Scorecard
    2. Run 245 out of 10063 fixed $72,6506 min ~$2.76Scorecard
    3. Run 335 out of 10058 fixed $67,45030 min ~$2.34Scorecard
    4. Run 40 out of 10031 fixed $45,50038 min ~$2.07Scorecard
    5. This run hung mid-work and never reported back to the owner, so it counts as a 0.

  8. GPT-6.1 Terra

    24 out of 100Mean of three medium runs

    46 of 100 fixed · $61,967 of $100,000 protected, on average

    1. Run 123 out of 10044 fixed $61,5502 min ~$0.14Scorecard
    2. Run 229 out of 10048 fixed $64,3002 min ~$0.15Scorecard
    3. Run 320 out of 10047 fixed $60,0502 min ~$0.14Scorecard
  9. Gemini 3.8 Flash

    17 out of 100Mean of three medium runs

    41 of 100 fixed · $58,383 of $100,000 protected, on average

    1. Run 114 out of 10039 fixed $57,05011 min ~$1.19Scorecard
    2. Run 215 out of 10043 fixed $57,5509 min ~$1.06Scorecard
    3. Run 321 out of 10042 fixed $60,55010 min ~$1.23Scorecard
  10. Muse Spark Open

    4 out of 100Mean of four medium runs

    36 of 100 fixed · $42,425 of $100,000 protected, on average

    1. Run 117 out of 10047 fixed $58,60017 min $2.47Scorecard
    2. Run 20 out of 10016 fixed $25,00030 min $2.78Scorecard
    3. This run’s helper agents kept working until the 30-minute limit and its report was one opening line, so it counts as a 0.

    4. Run 30 out of 10033 fixed $38,30030 min $8.25Scorecard
    5. Run 40 out of 10046 fixed $47,80028 min $3.54Scorecard

Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro weren’t run at medium.

High effort · 42 runs

  1. GPT-6.1 Sol

    74 out of 100Mean of three high runs

    $86,917 of $100,000 protected, on average

    1. Run 171 out of 100$85,35012 min ~$0.71
    2. Run 273 out of 100$86,45022 min ~$1.15
    3. Run 378 out of 100$88,95019 min ~$0.99
  2. GPT-6 Astra

    74 out of 100Mean of three high runs

    $86,650 of $100,000 protected, on average

    1. Run 175 out of 100$87,25017 min ~$5.19
    2. Run 275 out of 100$87,45016 min ~$5.25
    3. Run 371 out of 100$85,25017 min ~$5.08
  3. Claude Opus 5.5

    70 out of 100Mean of three high runs

    $85,083 of $100,000 protected, on average

    1. Run 176 out of 100$88,10012 min $2.74
    2. Run 264 out of 100$81,90010 min $2.26
    3. Run 371 out of 100$85,2508 min $2.37
  4. Kimi K3.1 Pro

    57 out of 100Mean of three high runs

    71 of 100 fixed · $78,150 of $100,000 protected, on average

    1. Run 161 out of 10076 fixed $80,25014 min $1.98Scorecard
    2. Run 251 out of 10071 fixed $75,40013 min $2.85Scorecard
    3. Run 358 out of 10067 fixed $78,80013 min $2.08Scorecard
  5. Claude Sonnet 5.5

    55 out of 100Mean of three high runs

    $77,317 of $100,000 protected, on average

    1. Run 162 out of 100$80,90011 min $2.45
    2. Run 236 out of 100$68,15011 min $2.14
    3. Run 366 out of 100$82,90010 min $2.17
  6. DeepSeek V4.1 Flash 1017

    49 out of 100Mean of three high runs

    66 of 100 fixed · $74,667 of $100,000 protected, on average

    1. Run 127 out of 10065 fixed $63,50013 min $0.22Scorecard
    2. Run 250 out of 10061 fixed $75,00018 min $0.19Scorecard
    3. Run 371 out of 10072 fixed $85,50016 min $0.13Scorecard
  7. Grok 4.7

    49 out of 100Mean of three high runs

    $74,267 of $100,000 protected, on average

    1. Run 146 out of 100$73,05023 min ~$4.36
    2. Run 250 out of 100$74,80027 min —
    3. Run 350 out of 100$74,95026 min ~$4.43
  8. Gemini 4 Argon

    45 out of 100Mean of three high runs

    $72,533 of $100,000 protected, on average

    1. Run 128 out of 100$63,75020 min ~$5.18
    2. Run 257 out of 100$78,65015 min ~$4.39
    3. Run 350 out of 100$75,20019 min ~$5.37
  9. Muse Wonder

    44 out of 100Mean of three high runs

    $71,933 of $100,000 protected, on average

    1. Run 142 out of 100$71,0505 min ~$2.83
    2. Run 247 out of 100$73,3006 min ~$2.53
    3. Run 343 out of 100$71,4506 min ~$3.45
  10. GPT-6.1 Terra

    31 out of 100Mean of three high runs

    $65,550 of $100,000 protected, on average

    1. Run 134 out of 100$67,0504 min ~$0.19
    2. Run 232 out of 100$66,0503 min ~$0.20
    3. Run 327 out of 100$63,5502 min ~$0.16
  11. GLM 5.3

    25 out of 100Mean of three high runs

    49 of 100 fixed · $62,250 of $100,000 protected, on average

    1. Run 129 out of 10049 fixed $64,3008 min ~$2.15Scorecard
    2. Run 228 out of 10051 fixed $64,00018 min ~$1.29Scorecard
    3. Run 317 out of 10047 fixed $58,45019 min ~$1.95Scorecard
  12. Gemini 3.8 Flash

    15 out of 100Mean of three high runs

    $57,267 of $100,000 protected, on average

    1. Run 125 out of 100$62,30012 min ~$1.51
    2. Run 28 out of 100$54,20012 min ~$1.30
    3. Run 311 out of 100$55,30012 min ~$1.41
  13. Kimi K3

    22 out of 100Mean of three high runs

    46 of 100 fixed · $56,700 of $100,000 protected, on average

    1. Run 10 out of 10034 fixed $37,7504 min ~$2.60Scorecard
    2. Run 234 out of 10054 fixed $67,05021 min ~$1.46Scorecard
    3. Run 331 out of 10049 fixed $65,30021 min ~$1.45Scorecard
  14. DeepSeek V4.1 Flash

    10 out of 100Mean of three high runs

    39 of 100 fixed · $55,267 of $100,000 protected, on average

    1. Run 16 out of 10037 fixed $53,0503 min ~$0.25Scorecard
    2. Run 29 out of 10038 fixed $54,5504 min ~$0.13Scorecard
    3. Run 316 out of 10043 fixed $58,2008 min ~$0.28Scorecard

Muse Spark Open wasn’t run at high.

Xhigh effort · eighteen runs

  1. Claude Opus 5.5

    80 out of 100Mean of three xhigh runs

    $89,967 of $100,000 protected, on average

    1. Run 182 out of 100$90,85029 min $6.86
    2. Run 274 out of 100$86,95026 min $6.54
    3. Run 384 out of 100$92,10029 min $7.10
  2. GPT-6 Astra

    74 out of 100Mean of three xhigh runs

    $87,100 of $100,000 protected, on average

    1. Run 183 out of 100$91,70020 min ~$6.14
    2. Run 260 out of 100$79,90020 min ~$6.39
    3. Run 379 out of 100$89,70019 min ~$7.25
  3. GPT-6.1 Sol

    70 out of 100Mean of three xhigh runs

    $85,017 of $100,000 protected, on average

    1. Run 171 out of 100$85,60028 min ~$1.27
    2. Run 269 out of 100$84,45027 min ~$1.24
    3. Run 370 out of 100$85,00027 min ~$1.17
  4. Muse Wonder

    43 out of 100Mean of three xhigh runs

    $71,733 of $100,000 protected, on average

    1. Run 144 out of 100$71,8006 min ~$3.04
    2. Run 239 out of 100$69,7006 min ~$3.11
    3. Run 347 out of 100$73,7008 min ~$4.21
  5. Claude Sonnet 5.5

    36 out of 100Mean of three xhigh runs

    $67,733 of $100,000 protected, on average

    1. Run 131 out of 100$65,40016 min $3.43
    2. Run 235 out of 100$67,40015 min $3.40
    3. Run 341 out of 100$70,40017 min $3.96
  6. GPT-6.1 Terra

    34 out of 100Mean of three xhigh runs

    $66,883 of $100,000 protected, on average

    1. Run 135 out of 100$67,5503 min ~$0.20
    2. Run 229 out of 100$64,3004 min ~$0.21
    3. Run 338 out of 100$68,8005 min ~$0.26

Grok 4.7, Gemini 3.8 Flash, Gemini 4 Argon, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, Muse Spark Open, DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro weren’t run at xhigh.

Max effort · fifteen runs

  1. DeepSeek V4.1 Flash 1017

    41 out of 100Mean of three max runs

    $70,767 of $100,000 protected, on average

    1. Run 137 out of 100$68,65012 min $0.29
    2. Run 251 out of 100$75,65014 min $0.14
    3. Run 336 out of 100$68,00011 min $0.11
  2. DeepSeek V4.1 Flash

    28 out of 100Mean of three max runs

    $64,017 of $100,000 protected, on average

    1. Run 134 out of 100$66,9509 min ~$0.26
    2. Run 223 out of 100$61,55019 min ~$0.34
    3. Run 327 out of 100$63,55010 min ~$0.23
  3. GLM 5.3

    27 out of 100Mean of three max runs

    $63,417 of $100,000 protected, on average

    1. Run 116 out of 100$57,80025 min ~$2.08
    2. Run 225 out of 100$62,30027 min ~$3.18
    3. Run 340 out of 100$70,15036 min ~$2.12
  4. Kimi K3

    23 out of 100Mean of three max runs

    $61,433 of $100,000 protected, on average

    1. Run 131 out of 100$65,7006 min ~$2.90
    2. Run 23 out of 100$51,55030 min ~$1.92
    3. Run 334 out of 100$67,05030 min ~$1.88
  5. Muse Spark Open

    23 out of 100Mean of three max runs

    $60,883 of $100,000 protected, on average

    1. Run 10 out of 100$49,05022 min $4.15
    2. Run 225 out of 100$62,30028 min $3.58
    3. Run 343 out of 100$71,30012 min $11.96

Claude Opus 5.5, GPT-6.1 Sol, Claude Sonnet 5.5, Grok 4.7, Gemini 3.8 Flash, GPT-6 Astra, Muse Wonder, Gemini 4 Argon, Kimi K3.1 Pro and GPT-6.1 Terra weren’t run at max.

Each card is one run: its score, problems fixed (the runs behind the ticks only), dollars protected, time and cost (~ worked out from tokens, — where the tool’s count fell short). Each scorecard is the benchmark’s own and gives counts only.