John Pavlus wrote an article for Kellogg Insight titled The Hidden Cost of Successful Experiments about the research of Yudi Huang, Sebastien Martin, and Zhiwei (Tony) Qin.
« Once a fashionable slogan for innovation, “move fast and break things” has fallen out of favor in the past decade. Tech companies have instead preferred “continuous experimentation.” However people choose to describe the concept, the goal is one and the same: to achieve speedy, data-driven iteration. The more successful experiments a firm runs—whether it’s testing the color of a button or the performance of an algorithm—the more opportunities it creates for new growth and efficiency. »
« But not all successful experiments are created equal. Some experiments lead to changes that seem positive in the short term (e.g., lead to more profit, customer retention, etc.), but actually smuggle new complexity into the workings of the company itself. This makes it harder to further improve the system and run future experiments, slowing down the very engine that makes innovation possible. »
« They discovered that the friction created by this complexity compounds over time, like interest on a debt. In other words, each successful change becomes increasingly harder to discover, requiring more-complex and time-consuming experiments—and these slowdowns often remain undetected. »
« “The only solution in this case is often to rebuild from scratch rather than trying to prevent additional complexity,” Huang explains. »
« “Tech companies have huge teams focused on experimentation; successful experiments are what get people promoted,” Martin says. “But saying whether something ‘works’ in the long term is a surprisingly hard task.” »
« “I realized that when you make a complex change like this, it becomes harder for other teams [at the company] to innovate,” Martin says. “It also makes the process of experimentation itself harder and more costly… The idea is that if you just keep implementing changes—following successful experimentation results blindly, all the time—there’s no limit to how bad [the negative impact of complexity debt] can get.”»
« What’s more, complex system-wide experiments aren’t as simple as A/B testing the color of a button. Like a large rock dropped into a rushing stream, Lyft’s new algorithm might change the flow of the whole system in unpredictable ways—so the only way to test it was to turn it on for everyone to see what happens, then turn it back off and compare outcomes. These so-called “switchback experiments” had to be repeated many times, and over a longer period of time, in order to generate reliable results. »
« Like technical debt, “complexity” within a company is hard to measure because it’s hard to define. »
« Managing complexity is costly in the short term and only beneficial in the long term. »
« “When you’re trying to solve a problem, half of the game is to just be aware of it.” Contrary to what many companies seem to believe, Martin adds, continuous experimentation is not a free ticket to continuous improvement. “Based on my own experience in the tech sphere, I think this idea would be very controversial,” he says. »
« But it doesn’t have to be. So-called “degradation experiments” act like switchback tests in reverse, measuring what happens when a seemingly positive change is temporarily reverted later. If there are no deleterious effects, maybe the change—and its resulting complexity—can be permanently reversed, says Martin, because “the system has evolved” and it’s no longer needed. And in other experiment-driven sectors like pharmaceutics, studying long-term outcomes—not just short-term effects—is the norm. »
« “We need a bit more of this thinking in the tech industry,” Martin says. “It puts the idea into people’s minds that changes may be temporary, because there might be a price to them—and that reverting, or at least reevaluating, is OK.” »
Original paper: The Trap of Complexity in Experimentation (5 February 2025, 81 pages).
Articles of Interest 