Unchecked AB Testing Destroys Everything it Touches

Every infuriating thing on the web was once a successful experiment. As an industry, we are blessed with the ability to do fast, lightweight AB testing, and we are cursing ourselves by misusing that to juice metrics in the short term.

Unchecked AB Testing Destroys Everything it Touches
Photo by Jason Dent / Unsplash
And the justifications for each of the compromises are myriad, from the apathetic to outright cynical: they have to shop somewhere, everyone’s doing it, so we have to do it, people shop for travel so infrequently they’ll forget, no one’s complaining.
There’s two big problems with this:
1) if you’re not looking at the long-term, you may be doing serious long-term damage and not know it, and you’ll spiral out of control
2) you’ll open the door to disruptive competition that you almost certainly will be unable to respond to as a practical matter
As product managers, as designers, as humans who care, it is our responsibility to never, ever present something like that. We need to be careful and think through the long-term implications of changes as part of the initial experiment design and include them in planning the tests.
If we present results of early testing, we need to clearly elucidate both what we do and don’t know
Unchecked AB Testing Destroys Everything it Touches
Every infuriating thing on the web was once a successful experiment. Some smart person saw Normal site: 1% sign up for our newsletterThrow a huge modal offering 10% off first order: +100% sign ups …