Back
All Resources

What is Multivariate Testing?

And why are so many marketers transitioning to it?
Dan Pantelo

Multivariate testing, which has only recently become commercially viable for marketers, is unlocking an unprecedented world of possibilities for the marketing industry. A lot of these possibilities are still yet unrealized or unexplored because the technology that enables it is still so new.

Testing is a universal part of every marketer’s job because no marketer/brand ever really knows if everybody else will like what they like.

Creative Pitch Scene in MadMen

In the Mad Men era, artists would sketch out ideas for ads (sometimes hundreds of versions) and show them to survey groups. People getting paid minimum wage would sit there in conference rooms rating the ideas of advertising creatives. This went on for many decades… until the internet happened.

The history of testing in advertising is well-chronicled because it’s one of our biggest challenges that we know the least about. We started with survey groups, graduated to A/B testing, and now we’re breaking open the next frontier; multivariate testing.

A lot of folks characterize multivariate testing as ‘A/B testing on steroids.’ If you think of it this way, I’m glad you’ve made it here. The two have nothing to do with each other in practice or in results.

People are still confused about multivariate testing (MVT) because it’s so new. However, marketers who understand and execute MVT generate immense value and have a steep edge over their competitors. Soon, this will become the norm and no longer yield a huge advantage, but until then there is a knowledge arbitrage opportunity.

Below is an explanation of multivariate testing in 4 levels of increasing difficulty;

Oversimplified (for my grandma):

Multivariate Testing is a way to test every possible combination of variables. Since we are able to measure how every variable works with other variables, we are finally able to understand exactly what people love or hate the most about our ads.

Simplified (for beginners):

We need to test and — I cannot stress this enough — learn.

For the first time, we are able to understand why our best ads work so well, rather than just which ad works the best (aka, what A/B testing tells us).

Creative is the most important element in advertising success. When small creative details are changed, like the background color or the font, there is a massive impact on performance.

If changing your background color from blue to purple, or if moving the logo from the bottom left to the bottom right will make 2x more people click the ad, wouldn’t you want to know that?

Multivariate tests are the only way we can get the type of data to give us these insights because it tests every possible combination of as many variables as we can think up. Yes this involves creating a LOT of ads — for example, let’s say you want to test the following:

  • 5 background colors
  • 2 version of your logo
  • 3 images of the product
  • 5 headline slogans in the image
  • 2 fonts for those headlines

To create every permutation of these variables for a multivariate test, that would be (5 x 2 x 3 x 5 x 2) 300 ads! Obviously, it would be impractical for us to hire creatives to make each of those by hand, then have someone upload/run all those ads individually, and then analyze all that data properly. This is why multivariate testing needs technology/automation in order to be accessible in terms of price and time.

Multivariate Testing Grid Example

It’s also surprisingly cheap to run this many ads as long as you stick to the right multivariate methodology.

Going by the example above, imagine if using logo version #2 increased performance by 80% on average, no matter what else it was combined with. Now we know we should always use logo version #2 in our creative!

This data can then be used to inform and guide our creative process when we make new ads.

Tactical (for professional marketers & creatives):

By understanding the underlying variables that make our ads succeed, we can confront one of our biggest natural adversaries — creative fatigue.

We experience it all the time and it’s the bane of our existence — that dreaded moment when our best performing ad (the one that’s been carrying the whole account) starts to trend down and stops performing. Then it’s back to square one — the search for another hit.

Creative fatigue is natural, but multivariate testing gives us insurance against it.

By understanding which variables were accountable for the ad’s success, we can continue to reproduce its success within future creative iterations. No more guesswork! Plus, now we know exactly what to tell creatives to make next.

This allows us to secure a consistent performance-increase journey as we continue to test and learn, rather than the unpredictable performance volatility we’ve become so accustomed to.

Granular (for analytics & data experts):

Multivariate testing enables the collection of a new type of data that allows marketers to approach creative as a science, rather than an art. We call this new type of data Creative Intelligence.

Creative Intelligence is the data we have about how consumers react to isolated visual variables. As we collect more of this type of data from the ground up across many different brands, we can unlock the potential to generate accurate creative prescriptions for what people will most want to see if we want them to perform a particular action. This will allow us to peak into the future without actually running any live tests. This can potentially enable us to explore hundreds/thousands of marketing possibilities within seconds, rapidly identifying and executing the biggest positive outlier.

Now that multivariate testing is commercially feasible thanks to new software, we are going to see a new breed of databases emerge with unprecedented capabilities — perhaps even scary ones.

It’s entirely possible that the continued development of multivariate testing technology will unlock the ability to accurately simulate consumer behavior in the near future by understanding how they react to visual stimuli, which can produce one of the most powerful and consequential systems ever made. It’s yet to be seen what experts and academics can do with this information.

Read our full creative testing guide

SIGN UP FOR THE EXPERIMENTAL METHOD NEWSLETTER
TO GET THE GUIDE

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.