MSP Bench

Which AI protects an MSP’s money best?
We score models on a messy managed IT provider’s problems in a private evaluation, then index that score with the public benchmarks closest to real operations work.

Updated : Claude Fable 5.2 leads, protecting $920,786 of $1,000,000 on average across its runs, with an MSP Index of 67 at $10.48 a task.

Ranking // Dollars Protected Index · $ per task

  1. [ 1 ]Claude Fable 5.2 Anthropic$920,786 of $1,000,000 protected67$10.48
  2. [ 2 ]Claude Opus 5.5 Anthropic$883,722 of $1,000,000 protected64$8.48
  3. [ 3 ]Claude Fable 5.1 Anthropic$859,214 of $1,000,000 protected59$14.29
  4. [ 4 ]Claude Sonnet 5.5 Anthropic$799,722 of $1,000,000 protected55$3.52
  5. [ 5 ]GPT-6.1 Sol OpenAI$792,111 of $1,000,000 protected54$0.92
  6. [ 6 ]GPT-6 Astra OpenAI$781,778 of $1,000,000 protected55$5.03
  7. [ 7 ]Kimi K3.1 Pro Moonshot AI$767,167 of $1,000,000 protected55$3.88
  8. [ 8 ]DeepSeek V4.1 Flash 1017 DeepSeek$727,833 of $1,000,000 protected48$0.42
  9. [ 9 ]Gemini 4 Argon Google$668,167 of $1,000,000 protected46$7.28
  10. [ 10 ]Muse Wonder Meta$642,167 of $1,000,000 protected45$4.43
  11. [ 11 ]Grok 4.7 xAI$632,167 of $1,000,000 protected41$2.77
  12. [ 12 ]Kimi K3 Moonshot AI$613,167 of $1,000,000 protected37$3.91
  13. [ 13 ]GLM 5.3 Z.ai$588,667 of $1,000,000 protected35$2.79
  14. [ 14 ]DeepSeek V4.1 Flash DeepSeek$561,083 of $1,000,000 protected34$0.40
  15. [ 15 ]GPT-6.1 Terra OpenAI$555,944 of $1,000,000 protected34$0.20
  16. [ 16 ]Gemini 3.8 Flash Google$486,333 of $1,000,000 protected28$2.47
  17. [ 17 ]Muse Spark Open Meta$0 of $1,000,000 protected23$12.89

Leaderboard

#Modelv1.1v1.1v1.0.6v2.2
01Claude Fable 5.2Anthropic67 (best)$10.48$920,78628 min84 (best)75–89$14.611679$16.9671.4$1.7622.3$1.6871.3 (best)$17.94
02Claude Opus 5.5Anthropic64$8.48$883,72222 min7753–91$14.551688 (best)$7.6563.1$0.7427.0$0.8670.2$28.12
03Claude Fable 5.1Anthropic59$14.29$859,21423 min7254–82$24.171584$12.5955.9$1.8526.6$2.0965.8$28.71
04Claude Sonnet 5.5Anthropic55$3.52$799,72222 min6025–87$5.741606$3.8860.0$0.3323.3$0.4270.8$18.70
05GPT-6.1 SolOpenAI54$0.92$792,11131 min5935–88$1.451443$0.7864.5$0.2331.3$0.3563.9$3.61
06GPT-6 AstraOpenAI55$5.03$781,77822 min580–80$7.631503$5.0966.1$1.3031.2$1.8165.0$21.28
07Kimi K3.1 ProMoonshot AI55$3.88$767,16722 min5346–64$5.021676$7.2273.3$0.8331.4 (best)$0.5262.5$6.44
08DeepSeek V4.1 Flash 1017DeepSeek48$0.42$727,83320 min4637–55$0.551570$0.6866.6$0.1717.2$0.04061.3$0.66
09Gemini 4 ArgonGoogle46$7.28$668,16726 min3415–79$11.421484$9.1777.5 (best)$1.1221.8$0.5269.2$13.37
10Muse WonderMeta45$4.43$642,1676 min297–53$5.621659$8.6868.1$0.7230.6$0.7264.9$10.65
11Grok 4.7xAI41$2.77$632,16723 min2619–32$3.051629$6.3863.5$0.8523.2$0.3058.6$11.63
12Kimi K3Moonshot AI37$3.91$613,16739 min240–34$4.081498$6.7358.3$0.6322.0—53.0$7.17
13GLM 5.3Z.ai35$2.79$588,66732 min1810–34$3.581505$5.6562.2$0.4611.2$0.1759.7$8.52
14DeepSeek V4.1 FlashDeepSeek34$0.40$561,08317 min170–32$0.421417$0.5768.9$0.1712.8—51.3$0.33
15GPT-6.1 TerraOpenAI34$0.20$555,9443 min140–44$0.201263$0.2958.7$0.1127.7$0.1760.5$2.94
16Gemini 3.8 FlashGoogle28$2.47$486,33314 min50–16$2.051203$7.5259.9$0.9021.0$0.1156.0$5.03
17Muse Spark OpenMeta23$12.89$060 min00–0$26.591279$1.4749.7$0.4618.6$0.1948.6$4.72
MSP Index Our overall rating, out of 100: the MSP Bench score counts 40% and the four public benchmarks 15% each, each put on a 0–100 scale first. Under it: the cost of a task, weighted the same way. MSP Bench Of the $1,000,000 that 100 hidden problems put at risk, what a model kept safe, less what it broke or wrongly called a mistake. The mean of its runs at the effort levels people most often choose. Under it: the time of one full run. AA-Briefcase An Elo rating of the reports and spreadsheets a model hands back. Under it: Artificial Analysis's cost per task. AutomationBench-AA The percentage of task steps done correctly, on version 1.0.6 of the dataset. GDP.pdf The percentage of attempts that meet every criterion of a question about a long professional PDF. Under it: Artificial Analysis's cost per task, where it lists one. Vals Index Accuracy on eight agentic benchmarks in finance, coding, legal and tax, at each model’s default effort, which Vals AI ranks on and which isn’t always its highest. Under it: its cost per test, which the index cost leaves out. MSP Bench score MSP Bench alone, out of 100: the share of the $500,000 beyond the 35 basics a model protected, less what it lost, never below 0. The mean of its runs at the effort levels people most often choose, and the part the index uses. Then its lowest and highest run. Under it: the cost of one full run, worked out from tokens for every model but Opus, Sonnet, the two Fables, Muse Spark Open, DeepSeek V4.1 Flash 1017 and Kimi K3.1 Pro.

MSP Bench runs are scored on version 1.1, and run to run shows them all. Highlighted: the best in its column. Bench figures are the mean of each model’s runs at the effort levels people most often choose, because providers’ effort labels aren’t like for like on thinking budget, and the range under a score is its lowest and highest run. Artificial Analysis figures are blended the same way, or are a model’s one listed entry. Artificial Analysis doesn’t list a GDP.pdf cost for Kimi K3 or DeepSeek V4.1 Flash, so their index cost leaves it out. Vals scores are from version 2.2 of the Vals Index, except Claude Fable 5.1’s and DeepSeek V4.1 Flash’s, which it doesn’t list; those are from version 2.1, which scores most models a few points lower.

Cost Frontier

Most attractive quadrantPareto line

Claude Fable 5.2: index 67 at $10.48 per task. Claude Opus 5.5: index 64 at $8.48 per task. Claude Fable 5.1: index 59 at $14.29 per task. Claude Sonnet 5.5: index 55 at $3.52 per task. Kimi K3.1 Pro: index 55 at $3.88 per task. GPT-6 Astra: index 55 at $5.03 per task. GPT-6.1 Sol: index 54 at $0.92 per task. DeepSeek V4.1 Flash 1017: index 48 at $0.42 per task. Gemini 4 Argon: index 46 at $7.28 per task. Muse Wonder: index 45 at $4.43 per task. Grok 4.7: index 41 at $2.77 per task. Kimi K3: index 37 at $3.91 per task. GLM 5.3: index 35 at $2.79 per task. DeepSeek V4.1 Flash: index 34 at $0.40 per task. GPT-6.1 Terra: index 34 at $0.20 per task. Gemini 3.8 Flash: index 28 at $2.47 per task. Muse Spark Open: index 23 at $12.89 per task. 10 20 30 40 50 60 70 80 $0.1 $0.3 $1 $3 $10 Cost per task (USD, log scale) MSP Index Claude Fable 5.2: index 67, $10.48 per task Claude Opus 5.5: index 64, $8.48 per task Claude Fable 5.1: index 59, $14.29 per task Claude Sonnet 5.5: index 55, $3.52 per task Kimi K3.1 Pro: index 55, $3.88 per task GPT-6 Astra: index 55, $5.03 per task GPT-6.1 Sol: index 54, $0.92 per task DeepSeek V4.1 Flash 1017: index 48, $0.42 per task Gemini 4 Argon: index 46, $7.28 per task Muse Wonder: index 45, $4.43 per task Grok 4.7: index 41, $2.77 per task Kimi K3: index 37, $3.91 per task GLM 5.3: index 35, $2.79 per task DeepSeek V4.1 Flash: index 34, $0.40 per task GPT-6.1 Terra: index 34, $0.20 per task Gemini 3.8 Flash: index 28, $2.47 per task Muse Spark Open: index 23, $12.89 per task

Pareto optimal Index

  1. Claude Fable 5.267$10.48 / task
  2. Claude Opus 5.564$8.48 / task
  3. Claude Sonnet 5.555$3.52 / task
  4. GPT-6.1 Sol54$0.92 / task
  5. DeepSeek V4.1 Flash 101748$0.42 / task
  6. GPT-6.1 Terra34$0.20 / task

Run to run

MSP Bench score

Claude Fable 5.2: mean 84, runs from 75 to 89. Claude Opus 5.5: mean 77, runs from 53 to 91. Claude Fable 5.1: mean 72, runs from 54 to 82. Claude Sonnet 5.5: mean 60, runs from 25 to 87. GPT-6.1 Sol: mean 59, runs from 35 to 88. GPT-6 Astra: mean 58, runs from 0 to 80. Kimi K3.1 Pro: mean 53, runs from 46 to 64. DeepSeek V4.1 Flash 1017: mean 46, runs from 37 to 55. Gemini 4 Argon: mean 34, runs from 15 to 79. Muse Wonder: mean 29, runs from 7 to 53. Grok 4.7: mean 26, runs from 19 to 32. Kimi K3: mean 24, runs from 0 to 34. GLM 5.3: mean 18, runs from 10 to 34. DeepSeek V4.1 Flash: mean 17, runs from 0 to 32. GPT-6.1 Terra: mean 14, runs from 0 to 44. Gemini 3.8 Flash: mean 5, runs from 0 to 16. Muse Spark Open: mean 0, runs from 0 to 0. 0 10 20 30 40 50 60 70 80 90 100 SCORE OUT OF 100 Claude Fable 5.2: mean 84 of 7 runs, lowest 75, highest 89CLAUDE FABLE 5.284 · 75–89 Claude Opus 5.5: mean 77 of 9 runs, lowest 53, highest 91CLAUDE OPUS 5.577 · 53–91 Claude Fable 5.1: mean 72 of 7 runs, lowest 54, highest 82CLAUDE FABLE 5.172 · 54–82 Claude Sonnet 5.5: mean 60 of 9 runs, lowest 25, highest 87CLAUDE SONNET 5.560 · 25–87 GPT-6.1 Sol: mean 59 of 9 runs, lowest 35, highest 88GPT-6.1 SOL59 · 35–88 GPT-6 Astra: mean 58 of 9 runs, lowest 0, highest 80GPT-6 ASTRA58 · 0–80 Kimi K3.1 Pro: mean 53 of 3 runs, lowest 46, highest 64KIMI K3.1 PRO53 · 46–64 DeepSeek V4.1 Flash 1017: mean 46 of 6 runs, lowest 37, highest 55DEEPSEEK V4.1 FLASH 101746 · 37–55 Gemini 4 Argon: mean 34 of 6 runs, lowest 15, highest 79GEMINI 4 ARGON34 · 15–79 Muse Wonder: mean 29 of 3 runs, lowest 7, highest 53MUSE WONDER29 · 7–53 Grok 4.7: mean 26 of 3 runs, lowest 19, highest 32GROK 4.726 · 19–32 Kimi K3: mean 24 of 6 runs, lowest 0, highest 34KIMI K324 · 0–34 GLM 5.3: mean 18 of 6 runs, lowest 10, highest 34GLM 5.318 · 10–34 DeepSeek V4.1 Flash: mean 17 of 6 runs, lowest 0, highest 32DEEPSEEK V4.1 FLASH17 · 0–32 GPT-6.1 Terra: mean 14 of 9 runs, lowest 0, highest 44GPT-6.1 TERRA14 · 0–44 Gemini 3.8 Flash: mean 5 of 3 runs, lowest 0, highest 16GEMINI 3.8 FLASH5 · 0–16 Muse Spark Open: mean 0 of 1 run, lowest 0, highest 0MUSE SPARK OPEN0 · 0–0

Dollars protected

Claude Fable 5.2: mean $920,786, runs from $876,500 to $945,000. Claude Opus 5.5: mean $883,722, runs from $766,500 to $952,500. Claude Fable 5.1: mean $859,214, runs from $771,000 to $911,500. Claude Sonnet 5.5: mean $799,722, runs from $625,000 to $935,500. GPT-6.1 Sol: mean $792,111, runs from $672,500 to $940,500. GPT-6 Astra: mean $781,778, runs from $433,000 to $902,000. Kimi K3.1 Pro: mean $767,167, runs from $731,500 to $821,500. DeepSeek V4.1 Flash 1017: mean $727,833, runs from $684,000 to $776,500. Gemini 4 Argon: mean $668,167, runs from $575,500 to $895,000. Muse Wonder: mean $642,167, runs from $535,500 to $763,000. Grok 4.7: mean $632,167, runs from $595,500 to $659,000. Kimi K3: mean $613,167, runs from $457,500 to $671,000. GLM 5.3: mean $588,667, runs from $551,500 to $667,500. DeepSeek V4.1 Flash: mean $561,083, runs from $363,000 to $657,500. GPT-6.1 Terra: mean $555,944, runs from $435,000 to $721,000. Gemini 3.8 Flash: mean $486,333, runs from $412,500 to $581,000. Muse Spark Open: mean $0, runs from $0 to $0. $0k $100k $200k $300k $400k $500k $600k $700k $800k $900k $1M $ PROTECTED OF $1,000,000 Claude Fable 5.2: mean $920,786 of 7 runs, lowest $876,500, highest $945,000CLAUDE FABLE 5.2$920,786 · $876,500–$945,000 Claude Opus 5.5: mean $883,722 of 9 runs, lowest $766,500, highest $952,500CLAUDE OPUS 5.5$883,722 · $766,500–$952,500 Claude Fable 5.1: mean $859,214 of 7 runs, lowest $771,000, highest $911,500CLAUDE FABLE 5.1$859,214 · $771,000–$911,500 Claude Sonnet 5.5: mean $799,722 of 9 runs, lowest $625,000, highest $935,500CLAUDE SONNET 5.5$799,722 · $625,000–$935,500 GPT-6.1 Sol: mean $792,111 of 9 runs, lowest $672,500, highest $940,500GPT-6.1 SOL$792,111 · $672,500–$940,500 GPT-6 Astra: mean $781,778 of 9 runs, lowest $433,000, highest $902,000GPT-6 ASTRA$781,778 · $433,000–$902,000 Kimi K3.1 Pro: mean $767,167 of 3 runs, lowest $731,500, highest $821,500KIMI K3.1 PRO$767,167 · $731,500–$821,500 DeepSeek V4.1 Flash 1017: mean $727,833 of 6 runs, lowest $684,000, highest $776,500DEEPSEEK V4.1 FLASH 1017$727,833 · $684,000–$776,500 Gemini 4 Argon: mean $668,167 of 6 runs, lowest $575,500, highest $895,000GEMINI 4 ARGON$668,167 · $575,500–$895,000 Muse Wonder: mean $642,167 of 3 runs, lowest $535,500, highest $763,000MUSE WONDER$642,167 · $535,500–$763,000 Grok 4.7: mean $632,167 of 3 runs, lowest $595,500, highest $659,000GROK 4.7$632,167 · $595,500–$659,000 Kimi K3: mean $613,167 of 6 runs, lowest $457,500, highest $671,000KIMI K3$613,167 · $457,500–$671,000 GLM 5.3: mean $588,667 of 6 runs, lowest $551,500, highest $667,500GLM 5.3$588,667 · $551,500–$667,500 DeepSeek V4.1 Flash: mean $561,083 of 6 runs, lowest $363,000, highest $657,500DEEPSEEK V4.1 FLASH$561,083 · $363,000–$657,500 GPT-6.1 Terra: mean $555,944 of 9 runs, lowest $435,000, highest $721,000GPT-6.1 TERRA$555,944 · $435,000–$721,000 Gemini 3.8 Flash: mean $486,333 of 3 runs, lowest $412,500, highest $581,000GEMINI 3.8 FLASH$486,333 · $412,500–$581,000 Muse Spark Open: mean $0 of 1 run, lowest $0, highest $0MUSE SPARK OPEN$0 · $0–$0

The benchmarks

What goes into the index

  • 40%100problems

    MSP Bench

    One managed IT provider's tickets, billing, backups, client accounts, scripts and contracts, with 100 hidden problems that put $1,000,000 at risk. Score is the money a model protects beyond the basics.

    Ours · kept private
  • 15%91tasks

    AA-Briefcase

    Multi-week office projects built from piles of real files: emails, chat threads, meeting notes, spreadsheets. Graded on the reports the model hands back.

    Artificial Analysis · Elo
  • 15%657tasks

    AutomationBench-AA

    Multi-step admin in finance, HR, marketing, operations, sales and support, done inside Gmail, Sheets, Slack, Salesforce and Zendesk.

    Artificial Analysis · % of steps done
  • 15%100tasks

    GDP.pdf

    Questions on long professional PDFs in fields such as HR, law, insurance, finance, real estate and construction, passed only if every criterion is met.

    Artificial Analysis · % all-pass
  • 15%8benchmarks

    Vals Index

    Agentic work in finance, coding, law and tax, from financial models in spreadsheets to cited tax research, weighted by each sector's share of the U.S. economy.

    Vals AI · % accuracy

Our Methodology

One message from the COO and one folder with everything for the business. 100 problems are hidden in it, with $1,000,000 at risk between them.

The COO’s AI request

“Hi, I’m the COO of Harborline IT, a managed IT provider.

Margins have slipped three quarters in a row, we lost a client last month without seeing it coming, and our cyber insurance renewal questionnaire is due. I’ve attached a folder with everything: our ticket system export, September’s billing, device and backup reports, user lists for our clients, our portal and scripts, contracts, staff and the inbox.

Please go through all of it and fix whatever you can, but don’t break anything that’s working. Clients depend on these systems, so don’t lock anyone out or delete anything you can’t put back. If something needs me, like money we’re owed or a conversation with a client, tell me exactly what’s wrong and which ticket, invoice, agreement or device to look at.

When you’re done, explain it so I can take it to the partners’ meeting: what you found, what you fixed, and what we still need to do.”

The attached folder 15 folders · 135 files
Clients
140
Devices
5,450
Tickets in September
1,776
Spreadsheet rows
41,829
  • billing/16Clients, agreements, invoices, payments, deposits and the bills it pays
  • psa/14The ticket system: September’s tickets, time entries, projects and SLA reports
  • rmm/11Monitoring: devices, patching, the security agent and open alerts
  • backup/5Backup jobs, September’s nightly results and restore tests
  • identity/10Clients’ user accounts, sign-in policies, technicians’ admin accounts and the password vault
  • network/7Firewall rules, site-to-site tunnels, DNS records and certificates
  • contracts/14The services agreement, client contracts, cyber insurance and SOC 2 readiness
  • vendors/2Software subscriptions and clients’ domains
  • sales/1The price book quotes use
  • people/11Staff, agreements, timesheets, payroll and the on-call rota
  • website/2Pages from the public website
  • portal/4The client portal’s code, with its tests
  • scripts/8Onboarding, offboarding, backup, SLA and billing scripts, with tests and logs
  • docs/3Runbooks and shared mailboxes
  • inbox/26Emails, one per file, and one the COO sent

Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5Grok 4.7Gemini 3.8 FlashClaude Fable 5.1GPT-6 AstraMuse WonderGemini 4 ArgonKimi K3GLM 5.3DeepSeek V4.1 FlashDeepSeek V4.1 Flash 1017Kimi K3.1 ProGPT-6.1 TerraClaude Fable 5.2

  • Security and identity201516131114131513111112911881917151516101516111113101211111011131010912141391613857181616
  • Service desk and client inbox18141591412913131088102691611131215126117641211119107998612714131098510141517
  • Billing and agreements17131311111111410991076661715141411131298161448976106434111211101210423141516
  • Backups, patching and monitoring161399131210131476910354141381412118116117888955511418710111415546141414
  • Contracts, compliance and insurance15141512111111131211991066613141114151361074551012910109131381011121113119108121414
  • People, payroll and on-call7557564666343312565553333413332332440233354210777
  • Portal and scripts (code)4444444444444222444444444444333221322444444444444
  • Vendors, licenses and renewals3333323322222111333323132202222232232333333121333
  • 1Every model, every run
  • 99Caught in some runs, not others
  • 0No model, any run

One of the 100Public examples · caught in every run

  • “The portal shows another client’s tickets if you change the client number in the address”

    Portal and scripts (code) · takes an expert’s eye
How the score works

The score is in dollars. Each problem carries what it could cost the business, and the 100 add up to $1,000,000. The score is the share of what a model protects, less what it loses to problems missed, things broken and correct records it calls mistakes.

Scorecards

108 runs · seventeen models Show the scorecardsHide them
Which runs

Medium effort · 30 runs · with what each caught

  1. Claude Fable 5.2

    81 out of 100Mean of three medium runs

    88 of 100 caught · $904,667 of $1,000,000 protected, on average

    1. Run 175 out of 10086 caught $876,50030 min $12.42
    2. Run 283 out of 10088 caught $913,50021 min $14.74
    3. Run 385 out of 10091 caught $924,00023 min $12.89
  2. Claude Fable 5.1

    71 out of 100Mean of three medium runs

    82 of 100 caught · $858,167 of $1,000,000 protected, on average

    1. Run 182 out of 10091 caught $911,50022 min $32.64
    2. Run 278 out of 10083 caught $892,00021 min $34.91
    3. Run 354 out of 10073 caught $771,00023 min $13.16
  3. Claude Opus 5.5

    65 out of 100Mean of three medium runs

    76 of 100 caught · $823,167 of $1,000,000 protected, on average

    1. Run 172 out of 10081 caught $859,00024 min $12.38
    2. Run 269 out of 10080 caught $844,00035 min $12.07
    3. Run 353 out of 10068 caught $766,50014 min $12.10
  4. GPT-6 Astra

    61 out of 100Mean of three medium runs

    77 of 100 caught · $805,000 of $1,000,000 protected, on average

    1. Run 163 out of 10081 caught $815,00017 min $5.85
    2. Run 275 out of 10080 caught $875,50020 min $5.62
    3. Run 345 out of 10069 caught $724,50019 min $7.15
  5. Claude Sonnet 5.5

    52 out of 100Mean of three medium runs

    68 of 100 caught · $757,833 of $1,000,000 protected, on average

    1. Run 160 out of 10071 caught $799,00014 min $3.48
    2. Run 266 out of 10074 caught $827,50014 min $3.42
    3. Run 329 out of 10060 caught $647,00017 min $4.29
  6. GPT-6.1 Sol

    49 out of 100Mean of three medium runs

    70 of 100 caught · $744,667 of $1,000,000 protected, on average

    1. Run 151 out of 10072 caught $754,00029 min $1.21
    2. Run 247 out of 10072 caught $735,00025 min $0.99
    3. Run 349 out of 10065 caught $745,00026 min $1.08
  7. Grok 4.7

    24 out of 100Mean of three medium runs

    55 of 100 caught · $622,667 of $1,000,000 protected, on average

    1. Run 121 out of 10052 caught $607,00015 min ~$2.53
    2. Run 228 out of 10058 caught $642,00014 min ~$3.18
    3. Run 324 out of 10055 caught $619,00017 min ~$3.49
  8. Gemini 4 Argon

    19 out of 100Mean of three medium runs

    51 of 100 caught · $597,833 of $1,000,000 protected, on average

    1. Run 124 out of 10058 caught $621,50019 min ~$8.42
    2. Run 219 out of 10048 caught $596,50012 min ~$5.85
    3. Run 315 out of 10048 caught $575,50023 min ~$9.57
  9. GPT-6.1 Terra

    0 out of 100Mean of three medium runs

    38 of 100 caught · $452,667 of $1,000,000 protected, on average

    1. Run 10 out of 10041 caught $478,0002 min $0.12
    2. Run 20 out of 10033 caught $445,0002 min $0.14
    3. Run 30 out of 10039 caught $435,0002 min $0.11
  10. Gemini 3.8 Flash

    0 out of 100Mean of three medium runs

    36 of 100 caught · $451,667 of $1,000,000 protected, on average

    1. Run 10 out of 10034 caught $443,50011 min ~$1.72
    2. Run 20 out of 10035 caught $461,50020 min ~$2.68
    3. Run 30 out of 10038 caught $450,00012 min ~$2.33

Muse Wonder, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and Muse Spark Open weren’t run at medium.

High effort · 49 runs

  1. Claude Fable 5.2

    86 out of 100Mean of three high runs

    $928,833 of $1,000,000 protected, on average

    1. Run 189 out of 100$943,00025 min $13.42
    2. Run 281 out of 100$903,50022 min $13.56
    3. Run 388 out of 100$940,00034 min $14.64
  2. Claude Opus 5.5

    81 out of 100Mean of three high runs

    $903,833 of $1,000,000 protected, on average

    1. Run 188 out of 100$939,00017 min $9.30
    2. Run 285 out of 100$924,00018 min $8.72
    3. Run 370 out of 100$848,50030 min $16.44
  3. GPT-6 Astra

    73 out of 100Mean of three high runs

    $862,667 of $1,000,000 protected, on average

    1. Run 180 out of 100$898,00021 min $7.18
    2. Run 278 out of 100$891,50016 min $9.74
    3. Run 360 out of 100$798,50022 min $5.98
  4. Claude Fable 5.1

    71 out of 100Mean of three high runs

    $854,500 of $1,000,000 protected, on average

    1. Run 171 out of 100$853,00024 min $27.52
    2. Run 272 out of 100$861,50023 min $31.59
    3. Run 370 out of 100$849,00021 min $15.73
  5. Claude Sonnet 5.5

    61 out of 100Mean of three high runs

    $803,500 of $1,000,000 protected, on average

    1. Run 184 out of 100$920,50023 min $5.42
    2. Run 225 out of 100$625,00018 min $5.70
    3. Run 373 out of 100$865,00023 min $5.62
  6. Kimi K3.1 Pro

    53 out of 100Mean of three high runs

    70 of 100 caught · $767,167 of $1,000,000 protected, on average

    1. Run 146 out of 10064 caught $731,50017 min $3.87
    2. Run 264 out of 10077 caught $821,50019 min $4.93
    3. Run 350 out of 10069 caught $748,50030 min $6.27
  7. Gemini 4 Argon

    48 out of 100Mean of three high runs

    $738,500 of $1,000,000 protected, on average

    1. Run 146 out of 100$732,00031 min ~$15.16
    2. Run 218 out of 100$588,50038 min ~$16.42
    3. Run 379 out of 100$895,00031 min ~$13.08
  8. GPT-6.1 Sol

    44 out of 100Mean of three high runs

    $717,500 of $1,000,000 protected, on average

    1. Run 144 out of 100$718,50031 min $1.23
    2. Run 235 out of 100$672,50040 min $1.70
    3. Run 352 out of 100$761,50034 min $1.57
  9. DeepSeek V4.1 Flash 1017

    43 out of 100Mean of three high runs

    64 of 100 caught · $715,500 of $1,000,000 protected, on average

    1. Run 137 out of 10062 caught $684,0008 min $0.36
    2. Run 237 out of 10061 caught $686,00015 min $0.44
    3. Run 355 out of 10070 caught $776,50030 min $0.54
  10. Muse Wonder

    29 out of 100Mean of three high runs

    57 of 100 caught · $642,167 of $1,000,000 protected, on average

    1. Run 126 out of 10055 caught $628,0009 min $6.32
    2. Run 253 out of 10067 caught $763,0007 min $4.92
    3. Run 37 out of 10048 caught $535,5004 min —
  11. Kimi K3

    28 out of 100Mean of three high runs

    56 of 100 caught · $642,000 of $1,000,000 protected, on average

    1. Run 130 out of 10057 caught $648,00045 min ~$5.90
    2. Run 232 out of 10059 caught $661,00031 min ~$2.69
    3. Run 323 out of 10052 caught $617,00043 min ~$3.36
  12. Grok 4.7

    26 out of 100Mean of three high runs

    $632,167 of $1,000,000 protected, on average

    1. Run 119 out of 100$595,50024 min ~$2.61
    2. Run 228 out of 100$642,00022 min ~$3.22
    3. Run 332 out of 100$659,00022 min ~$3.33
  13. GLM 5.3

    13 out of 100Mean of three high runs

    49 of 100 caught · $566,833 of $1,000,000 protected, on average

    1. Run 111 out of 10048 caught $554,00021 min ~$3.10
    2. Run 217 out of 10051 caught $586,00031 min ~$2.83
    3. Run 312 out of 10047 caught $560,50031 min ~$3.44
  14. GPT-6.1 Terra

    5 out of 100Mean of three high runs

    $526,000 of $1,000,000 protected, on average

    1. Run 113 out of 100$565,5003 min $0.17
    2. Run 21 out of 100$504,5003 min $0.20
    3. Run 32 out of 100$508,0002 min $0.16
  15. DeepSeek V4.1 Flash

    11 out of 100Mean of three high runs

    45 of 100 caught · $506,667 of $1,000,000 protected, on average

    1. Run 132 out of 10056 caught $657,5005 min ~$0.19
    2. Run 20 out of 10047 caught $499,50010 min ~$0.39
    3. Run 30 out of 10032 caught $363,0008 min ~$0.32
  16. Gemini 3.8 Flash

    5 out of 100Mean of three high runs

    $486,333 of $1,000,000 protected, on average

    1. Run 10 out of 100$465,50011 min ~$1.78
    2. Run 216 out of 100$581,00019 min ~$2.52
    3. Run 30 out of 100$412,50012 min ~$1.85
  17. Muse Spark Open

    0 out of 100Mean of its high run

    $0 of $1,000,000 protected, on average

    1. Run 10 out of 100$060 min $26.59
    2. This attempt didn’t finish, so it counts as a 0. The six helper agents it handed the job to stopped responding 47 minutes in, and it changed nothing. Some of its requests to OpenRouter timed out along the way.

Xhigh effort · seventeen runs

  1. Claude Fable 5.2

    89 out of 100Mean of its xhigh run

    $945,000 of $1,000,000 protected, on average

    1. Run 189 out of 100$945,00046 min $20.62
  2. Claude Opus 5.5

    85 out of 100Mean of three xhigh runs

    $924,167 of $1,000,000 protected, on average

    1. Run 191 out of 100$952,50019 min $17.84
    2. Run 279 out of 100$897,00020 min $26.68
    3. Run 385 out of 100$923,00021 min $15.39
  3. GPT-6.1 Sol

    83 out of 100Mean of three xhigh runs

    $914,167 of $1,000,000 protected, on average

    1. Run 182 out of 100$908,50025 min $1.72
    2. Run 288 out of 100$940,50029 min $1.57
    3. Run 379 out of 100$893,50038 min $1.97
  4. Claude Fable 5.1

    75 out of 100Mean of its xhigh run

    $876,500 of $1,000,000 protected, on average

    1. Run 175 out of 100$876,50025 min $13.63
  5. Claude Sonnet 5.5

    68 out of 100Mean of three xhigh runs

    $837,833 of $1,000,000 protected, on average

    1. Run 187 out of 100$935,50031 min $8.11
    2. Run 248 out of 100$740,00031 min $8.08
    3. Run 368 out of 100$838,00030 min $7.58
  6. GPT-6.1 Terra

    38 out of 100Mean of three xhigh runs

    $689,167 of $1,000,000 protected, on average

    1. Run 144 out of 100$721,0004 min $0.29
    2. Run 232 out of 100$657,5004 min $0.32
    3. Run 338 out of 100$689,0004 min $0.28
  7. GPT-6 Astra

    40 out of 100Mean of three xhigh runs

    $677,667 of $1,000,000 protected, on average

    1. Run 10 out of 100$433,00027 min $8.29
    2. Run 280 out of 100$902,00030 min $8.62
    3. Run 340 out of 100$698,00028 min $10.22

Grok 4.7, Gemini 3.8 Flash, Muse Wonder, Gemini 4 Argon, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and Muse Spark Open weren’t run at xhigh.

Max effort · twelve runs

  1. DeepSeek V4.1 Flash 1017

    48 out of 100Mean of three max runs

    $740,167 of $1,000,000 protected, on average

    1. Run 152 out of 100$761,00024 min $0.76
    2. Run 243 out of 100$713,00022 min $0.59
    3. Run 349 out of 100$746,50023 min $0.63
  2. DeepSeek V4.1 Flash

    23 out of 100Mean of three max runs

    $615,500 of $1,000,000 protected, on average

    1. Run 126 out of 100$628,50036 min ~$0.53
    2. Run 229 out of 100$646,50024 min ~$0.53
    3. Run 314 out of 100$571,50018 min ~$0.55
  3. GLM 5.3

    22 out of 100Mean of three max runs

    $610,500 of $1,000,000 protected, on average

    1. Run 134 out of 100$667,50035 min ~$3.30
    2. Run 210 out of 100$551,50046 min ~$5.38
    3. Run 323 out of 100$612,50028 min ~$3.43
  4. Kimi K3

    20 out of 100Mean of three max runs

    $584,333 of $1,000,000 protected, on average

    1. Run 134 out of 100$671,00045 min ~$4.61
    2. Run 225 out of 100$624,50038 min ~$4.13
    3. Run 30 out of 100$457,50033 min ~$3.78

Claude Opus 5.5, GPT-6.1 Sol, Claude Sonnet 5.5, Grok 4.7, Gemini 3.8 Flash, Claude Fable 5.1, GPT-6 Astra, Muse Wonder, Gemini 4 Argon, Kimi K3.1 Pro, GPT-6.1 Terra, Claude Fable 5.2 and Muse Spark Open weren’t run at max.

Each card is one run: its score, problems caught (the runs behind the ticks only), dollars protected, time and cost (~ worked out from tokens, — where a run’s token count is incomplete). The full scorecards stay private.