Back

How Netflix Uses Testing

To Make Every Decision The Right Decision
Brett Friedman

The Company

Netflix is your favorite streaming service. If you've missed their existence so far, you haven't entered their audience test criteria yet. But don't worry, you will. It's just a matter of time.

The Experiment Pt. 1

Netflix has a self-proclaimed "A/B testing culture: nearly every decision [they] make about [their] product and business is guided by member behavior observed in test."

Just a few examples of what they're testing: UI, algorithms, messaging, marketing, operations, infrastructure changes, title artwork, personalization algorithms, video encoding... the list goes on. In fact, they've built an entire architecture for their data scientists to enable quicker iteration. If you want to learn more about their many experiments, you can read more on their dedicated research blog.

For now, we'll focus on how they test their artwork displays for each film they present to members. Their main experimental question is 'how can we improve the CTR of the first glance when a member opens Netflix?' To set up the experiment, they had their creative team build multiple versions of each film in question. Each member would get a different artwork variant. Then they would measure CTR as well as aggregate play duration, fraction of plays with short duration, fraction of content viewed... in short, they looked for measures of engagement.

Netflix Movie Artwork

The Experiment Pt. 2

Then, they went multivariate. Rather than simply switching up the full artwork in an a/b/n test, they began labeling components (named Stable Identifiers) in their artwork and systematically varying their appearances. Next, they ran all the variants to determine which pieces were most effective, in addition to which images were the overall best performers.

The Results

Both tests were resounding successes. With the first film they tested, they found a 14% higher CTR for an image variant. With their multivariate tests, they discovered a winning title format, as well as a subtler truth, "images that have expressive facial emotion that conveys the tone of the title do particularly well." Netflix continues to test artwork, as well as billboards, trailers, montages, and, of course, ads with the same multivariate system, attempting to find the best assets for each title on any canvas.


The Application

Next time you flip on Netflix, ask a friend to do the same and see if you can spot differing artwork. You can learn how to run multivariate tests yourself using Marpipe, for free, with our experimentation guides.

The Team

Website:https://www.netflix.com/

Read our full creative testing guide

SIGN UP FOR THE EXPERIMENTAL METHOD NEWSLETTER
TO GET THE GUIDE

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.