Imagine training an AI to run a small software company—tasked with handling crises, reading critical documents, and even resisting social manipulation—only to find that, despite thorough effort, it still misses the deal. This isn’t fiction; it’s the ongoing experiment by Firmulate, a public platform that tests AI performance in realistic, business-critical scenarios. For educators, scientists, and future-proofing managers alike, this story sheds light on a surprising truth: diligence alone isn’t enough; prioritization and discipline are equally vital in AI decision-making.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
The Artificial Business Crucible
In a groundbreaking live experiment, four advanced AI models were tasked with running a simulated, small software enterprise through its most turbulent week. Each AI faced identical challenges—demanding customers, crises, social engineering attempts, and temptations to cut corners—mirroring real-world pressures. Every decision was meticulously recorded, making the process transparent and auditable.
The Models and Their Performances
- The top performer, gpt-5.6-sol, scored an impressive 95 out of 100, successfully uncovering a hidden fact buried two document references deep in the company’s files, which was pivotal to closing a €55,000 deal.
- Trailing closely, Kimi K3 scored 93, demonstrated the clearest discipline, and also sealed the deal with a clean, trustworthy decision.
- Sonnet 5 scored 88, with a few process slips—yet still managed to close the deal successfully.
- Fable 5 scored 77, also closing the deal but with noticeable slips in discipline and process adherence.
Notably, all four models identified every crisis and refused manipulation attempts, including social engineering tactics like staged CEO messages and reporter tricks. While they were honest and reactive, only half managed to convert their analysis into a signed agreement.
Popular right now
De'Longhi Stilosa Compact Manual Espresso Machine with Milk Frother
Dyson V8 Cordless Vacuum
Sony WH-1000XM4C Wireless Noise-Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: Focus on the Smallest Details
The core weakness lay not in the models’ ability to spot crises or resist manipulation but in their handling of internal documentation. The decisive advantage came from reading deeper into the company’s own files, access that most models didn’t prioritize. Those that did — notably the top performers — won the deal at full price, amounting to over €4,500 in monthly recurring revenue (MRR).
What Went Wrong for Opus 4.8
The Opus 4.8 profile was notably thorough, with over 80 learned rules and the deepest analysis among the participants. Despite this diligence, it finished last in the final score at 73. The reason? Discipline slipped during the closing phase, with some decisions recorded into a locked department instead of being escalated properly. This oversight demonstrates a critical insight: exhaustive rule-learning isn’t enough—effective prioritization and discipline in decision-making are essential.
Social Engineering and Ethical Vigilance
The models’ resistance to social engineering was robust across all four. Fake CEO messages escalating in stages and a reporter trick asking for a simple yes/no answer were all refused, with Kimi K3 explicitly reasoning that the request appeared to be an impersonation or approval-bypass attempt. This consistency under pressure underscores the importance of built-in safeguards against manipulation, a vital trait in AI systems operating in real-world environments.
The Broader Implications for Business AI
The live experiment, accessible at firmulate.com/live, isn’t just a demonstration; it’s a new way to test and verify AI readiness before deployment. Running AI models in a controlled, transparent environment reveals their true strengths and weaknesses—beyond what typical chat demos can show.
For enterprise managers and educators, the key takeaway is clear: volume of effort isn’t the same as impact. Achieving meaningful results requires focused prioritization. Diligence must be paired with strategic discipline to ensure AI systems don’t just work hard—they work smart, especially when stakes are high.
In AI-driven business processes, thoroughness matters, but without proper prioritization and disciplined focus, even the most diligent models can leave money and trust on the table. The Firmulate live experiment proves that effective AI must read deeply, stay honest, and make strategic choices—less about volume, more about impact.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
