Back

How Amazon Uses Reversible Decisions

To Learn Faster Than Their Competition
Brett Friedman

The Company

Amazon, according to Amazon, aims to be "Earth's most customer-centric company." They do that by selling every product that can be sold online and more on their jungle-like marketplace. What's non-obvious about how they do business is their unparalleled bias towards experimentation.

Just to name a few successes that came through their rigorous test-and-learn process:

  • 1-Click Buying
  • Mechanical Turk
  • Wish List
  • Kindle Direct Publishing
  • Subscribe-and-Save
  • Web Services (AWS)
  • Shopping Cart
  • AmazonFresh
  • Prime
  • Free Shipping

If that wasn't enough, here's an excerpt from Jeff Bezos' iconic 1997 Annual Letter to Shareholders:

"One area where I think we are especially distinctive is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment.

Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.

Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten.

We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs.

This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments."

And a one-liner to drive it home: "Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.”


The Experiment

Instead of zooming in on a single experiment, consider the process by which Amazon runs experiments.

After wasting millions on tiny tests with no logic behind them, they built an idea vetting process:

  1. Make sure your experiment is a Type 2 decision: Reversible
  2. Determine your hypothesis
  3. Determine your Value Proposition to Amazon
  4. Build an Experiment Group small enough to be fed by two pizzas (any larger creates inefficiency)

Amazon ensures that all members of experiment groups understand statistics and have strong data-driven mindsets, because much like Booking.com, the highest-paid person's opinion is not the most important nor the final vote.

What's unique to Amazon's experiments is their typical process of idea generation. They reverse-engineer from customer experience, considering every step of interaction with their many interfaces while focusing on fostering long-term satisfaction.

That being said, they learn just as much from their failures as they do from their wins. If you haven't heard of A9, Auctions, or Endless it's because none of them worked out. Bezos claims they were all "me too" ideas, copycats of other companies. However, Auctions evolved into the idea for third-party sellers and Endless evolved into the Zappos acquisition.  


The Results

Like fellow FAANG stocks Apple, Netflix, and Google, Amazon crushes its competition, consistently expanding deeper into the online retail world through experimentation. With $11.5 Billion in revenue, 19 Wiki-worthy subsidiaries, and over a million employees, they do fairly well, to say the least.

With current projects (that we know of) ranging from Economics to Robotics, Amazon is forging its way towards a data-driven future.

The Application

Even if your entire team is only Two Pizzas large, you can still apply Amazon's decision criteria to your experiments. Try focusing on reversible decisions that have a chance to "score 1,000 runs" with one swing of the bat.

Read our full creative testing guide

SIGN UP FOR THE EXPERIMENTAL METHOD NEWSLETTER
TO GET THE GUIDE

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.