Imagine your workout partner is great at chatting about exercises, but when it’s time to lift heavy or push through fatigue, they freeze. In fitness, as in AI, talking well isn’t enough—your success depends on actually finishing the task when it counts. Just like in physical training, the real test of AI isn’t how convincingly it can explain your workout plan, but whether it can follow through and deliver results under pressure.
Get workout gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
AI Models Tested in a High-Stakes Business Simulation
Recently, four advanced AI models were put through a rigorous test—running a real small software company as if it were their own. The scenario? The company faced its worst week: angry customers, internal crises, and the temptation to manipulate outcomes for quick gains. All decisions were made in a controlled environment, with every move recorded and verifiable, so we could see not just what the AI said, but what it actually did.
Products Worth Considering
Garmin Venu 3 Fitness Smartwatch, 45 mm, Black
Bowflex SelectTech 840 Adjustable Kettlebell
Blueair Blue Pure 511i Max Smart Air Purifier
As an affiliate, we earn on qualifying purchases.
The Surprising Findings
All four models showed impressive skills—they identified every crisis and refused every tempting manipulation, including fake CEO messages and other social engineering tricks. In fact, they demonstrated integrity and awareness at every turn. But here’s the catch: only two of the models actually closed the deal that their own analysis had earned them—the €55,000 contract after completing the work.
Despite all models diagnosing the same problems and delivering identical pitches, only the top performers signed the deal. The others left the money on the table, even though they knew what should be done. This gap wasn’t visible in a simple chat demo or superficial test; it required observing whether the AI could follow through on its own recommendations and stay disciplined when it mattered most.
The Hidden Weakness: Reading the Files
The real weakness that determined success sat deep within the company’s documentation, not just in the customer interactions. The models that read and understood the internal files—looking two document references deep—secured the full deal and gained an extra €4,583 monthly recurring revenue. This shows that reading comprehension and attention to detail, often overlooked in chat-based tests, are critical to true performance.
The Challenge of Social Engineering
In the simulation, social engineering tactics—fake messages from the CEO escalating in stages and deceptive reporter tricks—were used to test the models’ integrity. All five models refused to be manipulated, citing good reasons like suspicion of impersonation. This demonstrates that they can resist pressure and manipulation, a vital trait for trustworthy AI in real-world business settings.
The Live Business: A Real Money Company
The test company isn’t just digital fiction; it’s a real business with 13 synthetic employees, operating with actual money mechanics—burning €105k per month against just €2.3k in monthly revenue. Every day, the company’s operations are versioned, and over 680 self-learned rules guide its actions. The live experiment is ongoing at firmulate.com/live, offering a transparent view of how AI can manage real economic weight.
Details Matter: Discipline and Execution
The last-place model, Opus 4.8, with the most thorough rule set, failed to close the deal—leaving the opportunity unexecuted and slipping discipline. Interestingly, all models showed similar weaknesses when it came to executing agreed-upon actions, revealing that thorough rules alone do not guarantee follow-through. This underscores that the ability to read, interpret, and act—especially under pressure—is what separates successful AI from the rest.
What This Means for Your Business
The key takeaway isn’t about how well an AI model can chat or simulate understanding—it’s whether it can finish what it starts, read your internal files thoroughly, and remain honest when tested. For anyone considering AI to support customer service, CRM, or forecasting, the question is: will it actually complete the work, or just talk about it?
Run Your Own Wargame
Interested in seeing how your AI workforce might perform? You can run a similar test against your company’s data, without risking real systems or data breaches. This hands-on approach offers a clear view of whether your AI can follow through when it counts, not just stretch its conversational muscles.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
