Chapter 2

2.Law: The Scientific Method, Not Silver Bullets

Big idea: Conversion rate optimization isn’t a “one and done” guessing game; it’s a scientific process based on iteration.

The book (and film)Moneyball tells the true story of how the second poorest team in baseball, the Oakland Athletics, won more regular season games than all but one of theirtwenty-nine competitors. They became one of the most successful franchises in MLB, and they did it with a payroll less than a third of the richest team—a meager $40 million compared to the New York Yankees’ $126 million.17 It was unprecedented. And it smashed the prevailing rule in baseball that the teams with the most money always win. How did they pull off this incredible winning streak?

They did it by rethinking baseball. Traditionally, baseball teams evaluated players on a few key metrics—batting average, home runs, RBIs, and so on—and a lot of gut assessment. Literally. The film pokes fun at player reviews such as “passes theeye-candy test,” while the book notes scouts’ language around players’ bodies sounded more like sports car talk than recruiting.

These unscientific evaluation tactics only “worked” for the richest teams who had piles of cash to throw at flashy players. The Oakland A’s, with theirbottom-of-the-barrel payroll, didn’t have that financial luxury, so they needed to find another way. And they did. When they stepped back and looked at the game (in what the book calls “a systematic scientific investigation of the sport”), they determined wins = getting players on base. So instead of pursuing big hitters who cost half their payroll, they went after players who could get on base, even if the players’ methods werenon-sexy, such as walks and getting hit by pitches.

They went after meaningful stats instead of pretty faces. This scientific approach worked. The Oakland A’s started winning like crazy and kept winning like crazy. Their former general manager (and now EVP) is a living legend because of it. The lesson? Scientific thinking and repeated small achievements (like getting on base) can lead to big results over time.

So what does this story have to do with conversion rates ine-commerce? Well, most brands are stuck in the “old” approach to conversion rate optimization. They think of conversion rate optimization in terms of silver bullets and a “guess and test” approach. They throw a lot of money at pretty players (big redesigns), make gut decisions about improvements (“it’s anice-looking carousel”), and expect both approaches to work.

The only problem is, they don’t. Not reliably and not in ways that sustainably improve revenue. Meanwhile, the brands doing something different are the ones pulling ahead and redefining the conversion rate optimization game.

Three Big Misconceptions about Conversion Rate Optimization

Most misconceptions around conversion rate optimization boil down to this: there’s aone-and-done way to approach it. Usually, this misunderstanding wears one of three outfits: (1) there’s a single “good” conversion rate (or benchmark), (2) a redesign will solve conversion problems, or (3) a big test will deliver all t