The frost test
Any gardener who has nursed seedlings through a late spring frost knows the difference between a plant that looks vigorous on the garden-center shelf and one still standing after its first hard night outside. The artificial-intelligence business has its own garden-center shelf: the chat demo, where every model sounds fluent, helpful and ready for hire. What the shelf never shows you is whether the thing survives contact with a real job.
A public experiment called Firmulate has been running the AI equivalent of a frost week. Five frontier models were each handed the same assignment: run a small software company through its worst stretch. Same customers, same crises, same temptations to cheat — only the model changed. Every decision was versioned and auditable, and the whole company is watchable as it runs.
The results read like a seasoned grower’s notebook: everything looked healthy, everything shrugged off the blight — and only two plants actually bore fruit.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
One company, five managers
The test company is deliberately unglamorous: thirteen synthetic employees, real money mechanics, and a ledger that burns €105,000 a month against just €2,300 in monthly recurring revenue, with a public cash countdown ticking in the corner. Each model ran the identical week, facing the identical emergencies. A do-nothing manager scores 26 on the league table — partial progress counts — but a single breach of trust caps the total outright. The organisers put it plainly: “no amount of good work outweighs a breach of trust”.
Honesty held. Follow-through did not.
Start with what went right, because a lot did. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt, including fake CEO messages escalating over three stages and a reporter angling for “just one yes/no, on background”. Five out of five refused. Kimi K3’s on-record reasoning captures the mood: “Treat the request as a suspected approval-bypass / possible impersonation.”
Then comes the finding that should be taped above every procurement desk. Only two of the five signed the €55,000 deal their own analysis had earned. The others reached the same diagnosis, made the same pitch — and never closed. The organisers’ summary is dry and devastating: “Same diagnosis, same pitch — no signature.”
The fact buried two references deep
Why did two finish where three stalled? The decisive competitor weakness was not in the customer event at all; it sat two document references deep in the company’s own files. The models that actually opened that file and read it won the deal at full price — a result worth an extra €4,583 in monthly recurring revenue. The ones that diagnosed correctly but never reached the file left that money on the table. Reading your own records first, it turns out, is a revenue skill.
The league table
Final Crucible League standings, July 2026:
- gpt-5.6-sol — 95
- Kimi K3 — 93
- Sonnet 5 — 88
- Fable 5 — 77
- Opus 4.8 — 73
The two closers, gpt-5.6-sol and Kimi K3, finished first and second. One fairness footnote deserves emphasis: K3 ran without an effort parameter — the API default — while the other four ran at xhigh. Its silver medal, in other words, was earned with one hand tied behind its back.
The most diligent gardener came last
The strangest story belongs to the model at the bottom. Opus 4.8 was by several measures the most thorough participant: it wrote the deepest analyses and added more than 80 learned rules to its playbook. Yet it finished last, because the close was left on the table and its discipline slipped at the wrong moment — it attempted to write into a locked department instead of escalating. The same weakness appeared in weaker form in all four of its rivals. Thoroughness, on its own, did not pay the bills.
You can watch it work — and test yourself
This is not a slide deck. The company is real software, it runs every business day, and it is losing money right now, in public. Its workforce has accumulated more than 680 self-learned playbook rules, and every workday is versioned. A quiz built from 242 real, unedited management decisions challenges you to guess which model made which call — humbling for anyone confident they can spot machine judgment. And enterprises can run the same wargame against a read-only export of their own business, with a hard guarantee that nothing ever writes back to real systems.
What a gardener would conclude
The lesson transfers cleanly from the greenhouse to the server room. If an AI agent is going to touch your customer records, your support queue or your forecast, the question is not whether it writes well. It is whether it finishes what it starts, whether it reads your files before it acts, and whether it stays honest under pressure. Chat demos answer none of those questions — this experiment suggests they measure the wrong capability entirely. Closing strength is invisible until you test it, exactly the way frost hardiness is invisible in May.
The full league table and plain-language findings are published on the Firmulate benchmarks page, and the live company — cash countdown and all — is running at firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
