Beta

How Booking.com Uses 1000's of Experiments

To Build a Culture of Scientists
Brett Friedman

The Company

Booking.com is the world’s largest price comparison site for travel expenses such as flights, hotels, car rentals, excursions, and more, offering discounts and rewards for consistent usage.

The Experiment Pt. 1

According to Lukas Vermeer, Director of Experimentation, they run over 1000 experiments at a time, at all times. More impressive, they run tests for each individual website visitor, in order to get specific learnings. This means, assuming each test is an A/B test, they have 2^1000 experiments active right now!

Rather than putting all their tests in one basket, they diversify by setting up sub-tests for each to find the perfect sequence of listings for each user, figuring out their preferences along the way.

The Experiment Pt. 2

In order to maintain such a high number of experiments, Booking.com's team needs to have the right mindset. Stuart Frisby, Booking's Ex-Director of Design, once set a few guidelines for their culture:

  • No HIPPOs (highest paid person's opinions)
  • Every decision is a democracy, but test every decision
  • Trust your tools

In addition to having a designer on every development team and a forced role change every 10-12 months, employees understand the business from a holistic perspective, making them more able to come up with new ideas, hypotheses, and, of course, tests. Then it's all up to your software partners to run experiments i.e. VWO, Optimizely, Mailchimp, or Marpipe ;D

The Results

Booking.com was born in 1996. In 2005, Priceline bought them for $133mm. By 2011, they had net profits over $1.1 Billion. Today they make nearly $5 Billion in profit and publicly trade as BKNG, after acquiring most of their largest competitors (Kayak.com, OpenTable, momondo, etc.).

Their results are clear. Experimentation breeds success in business.  

The Application

While adding a designer to every dev team and avoiding centralized executive decision-making may be hard for companies with fewer employees than Booking's 26,000+ strength staff, one guideline is easy to follow; it's something we say all the time at Marpipe:

Test Everything!

The Team


Website: https://www.booking.com/

Boost ad performance in days with a 7 day free trial.
Claim Trial

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Outliers is the weekly newsletter that over 10,000 marketers rely on to get new data and tactics about creative testing.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Sign Up to Outliers 💌

Join 7,000+ marketers in the world’s best newsletter all about catalog ads!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.