How MassMutual and Mass General Brigham turned AI pilot sprawl into production results

Enterprise AI programs rarely fail because of bad ideas. More often, they get stuck in ungoverned pilot mode and never reach production. At a recent VentureBeat event, technology leaders from MassMutual and Mass General Brigham explained how they avoided that trap — and what the results look like when discipline replaces sprawl.

At MassMutual, the results are concrete: 30% developer productivity gains, IT help desk resolution times reduced from 11 minutes to one, and customer service calls cut from 15 minutes to just one or two. Sears Merritt, MassMutual’s head of enterprise technology and experience, spoke at the event about the company’s approach to AI implementation. “We’re always starting with why do we care about this problem?” Merritt said. “If we solve the problem, how are we gonna know we solved it? And, how much value is associated with doing that?”

Defining metrics and establishing strong feedback loops were key to MassMutual’s success. The company, a 175-year-old firm serving millions of policy owners and customers, has pushed AI into production across various business areas, including customer support, IT, customer acquisition, underwriting, servicing, claims, and more. Merritt explained that his team follows the scientific method, beginning with a hypothesis and testing whether it has an outcome that will tangibly drive the business forward.

Some ideas may be great, but they may be “intractable in the business” due to factors like lack of data or access, or regulatory constraints. “We won’t go any further with an idea until we get crystal clear on how we’re going to measure, and how we’re going to define success,” Merritt emphasized. Ultimately, it’s up to different departments and leaders to define what quality means: Choose a metric and define the minimum level of quality before a tool is placed into the hands of teams and partners. That starting point creates a quick feedback loop.

“The things that we find slow us down is where there isn’t shared clarity on what outcome we’re trying to achieve,” which can lead to confusion and constant re-adjusting, Merritt noted. By focusing on clear goals, measurable outcomes, and disciplined execution, MassMutual has been able to transform AI pilot projects into production results, delivering significant improvements in efficiency and customer satisfaction.

Similarly, Mass General Brigham, a major healthcare provider, has also leveraged AI to enhance its operations. The organization has implemented AI-driven tools to improve patient care, streamline administrative processes, and optimize resource allocation. By adopting a structured approach to AI implementation, Mass General Brigham has avoided the pitfalls of ungoverned pilots and achieved meaningful results.

In conclusion, the success stories of MassMutual and Mass General Brigham highlight the importance of discipline, clear goals, and measurable outcomes in AI implementation. By following a methodical approach that combines hypothesis testing, metric definition, and strong feedback loops, these organizations have transformed AI from pilot projects into production tools, driving tangible business value and improving customer experiences. As AI continues to evolve, these lessons will be crucial for other enterprises seeking to harness the full potential of AI without getting stuck in ungoverned pilot mode.