How Uses 1000's of Experiments

To Build a Culture of Scientists
Brett Friedman

The Company is the world’s largest price comparison site for travel expenses such as flights, hotels, car rentals, excursions, and more, offering discounts and rewards for consistent usage.

The Experiment Pt. 1

According to Lukas Vermeer, Director of Experimentation, they run over 1000 experiments at a time, at all times. More impressive, they run tests for each individual website visitor, in order to get specific learnings. This means, assuming each test is an A/B test, they have 2^1000 experiments active right now!

Rather than putting all their tests in one basket, they diversify by setting up sub-tests for each to find the perfect sequence of listings for each user, figuring out their preferences along the way.

The Experiment Pt. 2

In order to maintain such a high number of experiments,'s team needs to have the right mindset. Stuart Frisby, Booking's Ex-Director of Design, once set a few guidelines for their culture:

  • No HIPPOs (highest paid person's opinions)
  • Every decision is a democracy, but test every decision
  • Trust your tools

In addition to having a designer on every development team and a forced role change every 10-12 months, employees understand the business from a holistic perspective, making them more able to come up with new ideas, hypotheses, and, of course, tests. Then it's all up to your software partners to run experiments i.e. VWO, Optimizely, Mailchimp, or Marpipe ;D

The Results was born in 1996. In 2005, Priceline bought them for $133mm. By 2011, they had net profits over $1.1 Billion. Today they make nearly $5 Billion in profit and publicly trade as BKNG, after acquiring most of their largest competitors (, OpenTable, momondo, etc.).

Their results are clear. Experimentation breeds success in business.  

The Application

While adding a designer to every dev team and avoiding centralized executive decision-making may be hard for companies with fewer employees than Booking's 26,000+ strength staff, one guideline is easy to follow; it's something we say all the time at Marpipe:

Test Everything!

The Team


Read our full creative testing guide


Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.