By Tycho Ferrigni · September 5, 2025 · 6 min read
According to MIT, 95% of investments in GenAI have produced zero measurable returns… This is from the Harvard Business review this week and the title of the paper is “Beware the AI Experimentation Trap.” Here is the link.
We’ve hinted, or more accurately, danced around this topic in a couple of previous blog posts. The paper does a good job of pushing business leadership to get back to the basics of software development. If I had to summarize the entire paper into one sentence: “Get laser focused on creating Customer Value with any GenAI experimentation.” The shotgun approach of trying a bunch of things to see if something…anything???…works has probably run its course with GenAI.
The good news is that as long as your teams have a customer centric outcome that impacts the customer, your GenAI work has a reasonable chance of succeeding. With 95% of investments in GenAI not generating a measurable return, this seems like sage advice. So…if you can’t tell immediately what outcome is going to be different when your GenAI project is done, lightly tap the breaks, pull over to the side of the road and check your (road)map.
We don’t want any of you to think we don’t like GenAI. We absolutely love it. It has helped us become way more productive. That said, it’s going to be a bit (still thinking years) before we have true human replacement level intelligence from an algorithm. So…given the usefulness of the tool, how do we build things that create value for business? Same as we always have, focus on customer outcomes first…
It is our hope that you all get a chance to read the article. We asked GenAI to summarize the article (see below)… and it’s a pretty good synapsis. If you hop down to Framework for Productive AI Transformation #2 is great advice.
Here’s the GenAI summary…
Context & Concern
- A new MIT Media Lab/Project NANDA report found that 95% of investments in generative AI (gen AI) have produced zero measurable returns.
- This has fueled skepticism that AI is overhyped, especially with the underwhelming launch of GPT-5 and Gartner’s claim that gen AI is entering the “trough of disillusionment” in its hype cycle.
- The nuance: while individuals use AI productively, companies struggle to realize enterprise-level ROI, especially since most investment has gone to sales/marketing rather than back-end transformations (where ROI is higher).
Core Argument
- The problem is not that experimentation is bad, but that experimentation is often unfocused and disconnected from real business opportunities.
- This mirrors mistakes made in the earlier digital transformation era, when companies let “10,000 flowers bloom” but lacked clear ties to customer value, resulting in wasted effort.
- To avoid repeating that cycle, AI initiatives need direction, connection to customer value, and a plan for scaling.
Framework for Productive AI Transformation
- Understand AI in the Larger Arc of Transformation
- AI is part of the broader shift from digital at the periphery (IT tools, databases) to digital at the core (data- and AI-driven workflows).
- Every company is becoming a technology company, where humans oversee but don’t directly perform most operational decisions.
- The purpose of AI is not disruption for its own sake, but serving customers better, faster, cheaper.
- Focus on Serving Customers Better
- Don’t chase headlines about radical AI disruption or imitate tech giants.
- Use AI to improve internal operations and customer journeys.
- Identify value-creating “Trojan Horse” opportunities—near-term projects that prove value and build organizational learning.
- Run Focused Experiments with an Eye to Scaling
- Good experiments meet three conditions:
a) Connected to real value creation.
b) Low-cost, enabling multiple cycles of iteration.
c) Designed with scalability in mind.
- Scale Up with a Ninja Team
- Moving from pilot to enterprise scale requires dedicated scaling teams with leadership backing, resources, and company-wide reach.
- Examples include Amazon, Qualtrics, and 7-Eleven, where “ninja teams” drove scaling.
Conclusion
- We are past the AI hype wave, and many leaders risk misinterpreting implementation challenges as proof that AI cannot deliver value.
- In reality, AI can generate significant returns (e.g., multi-agent systems in back-end ops), but only if companies avoid unfocused experimentation and instead align initiatives with customer problem-solving.
- The fundamental truth: technology changes, but the purpose of business remains constant—solving important problems for customers.
