Why multivariate testing is the new standard in advertising

Testing is a universal part of every marketer’s job because no marketer or brand team ever really knows if everybody else will like what they like. The history of ad testing methods is well-chronicled because it’s one of our biggest challenges that we know the least about.

Multivariate testing — or MVT for short — has only recently become commercially viable for marketers, thanks to automated software. This testing approach is especially powerful when applied to ad creative because it not only tells you which ad people like best, but also which parts of that ad people like best. Multivariate testing arms marketers with the micro-level data they need to boost ad performance and conversion rates.

What is multivariate testing?

People are still wrapping their heads around multivariate testing because it’s so new. But, marketers who understand and run multivariate tests have a steep edge over their competitors.

Here are four different explanations of multivariate testing, each with an increasing level of technical detail.

Oversimplified (for my grandma):

Multivariate testing is a way to test every possible combination of variables. Since we are able to measure how every variable works with other variables, we are finally able to understand exactly what people love or hate the most about our ads.

Simplified (for beginners):

Multivariate testing lets marketers and advertisers test and  — most importantly —  actually learn.

For the first time, we are able to understand why our best ads work so well, rather than just which ad works the best (which is all A/B testing tells us).

When small creative details are changed, like the background color or the font, there is a massive impact on performance. We can use this information to affect conversion rate optimization.

If changing your background color from blue to purple or moving the logo from the left to the right would make twice as many people click your ad, wouldn’t you want to know that?

Multivariate tests are the only way we can get this level of data and meaningful results because it tests every possible combination of as many variables as we can think up. Yes, this involves creating a LOT of ads . For example, let’s say you want to test the following:

• 5 background colors
• 2 calls to action
• 3 product images
• 5 headlines
• 2 fonts for those headlines

To create every permutation of these variables for an MVT test, that would be 5 x 2 x 3 x 5 x 2 — for a total number of 300 ads! It would be impractical for us to hire creatives to make each of these testing variations by hand, have someone upload and run all those ads individually, and then analyze all of that data properly. This is why multivariate testing requires automation and a specific design approach to keep production time and costs low.

‍

‍

It’s also surprisingly cheap to run this many ads, as long as you stick to the right multivariate methodology.

Going by the example above, imagine if using the lime green text box increased performance by 80% on average, no matter what else it was combined with. This data can be used to inform and guide our creative process when we make new ads.

Tactical (for professional marketers and creatives):

By understanding the underlying variables that make our ads succeed, we can confront one of our biggest natural adversaries  —  creative fatigue.

We experience it all the time and it’s the bane of our existence : that dreaded moment when our best-performing ad (the one that’s been carrying the whole account) starts to trend down and stops performing. Then it’s back to square one  —  the search for another hit.

Creative fatigue is natural, but multivariate testing gives us insurance against it.

By understanding which variables were accountable for the ad’s success, we can continue to reproduce its success within future creative iterations. No more guesswork — we know exactly what to tell our creative team to design next.

This test-and-learn cycle of finding winning elements and iterating on them further creates a more consistent performance-increase journey. We can finally get off of the unpredictable performance rollercoaster we’ve become so used to.

Granular (for analytics and data experts):

Multivariate testing enables the collection of a new type of data that allows marketers to approach creative as a science, rather than an art. We call this new type of data creative intelligence.

Creative intelligence is the information we have about how consumers react to isolated creative variables — a headline, an image, a button color. As we collect more of this type of data from the ground up across many different brands, we have the potential to generate accurate creative prescriptions for what people will most want to see if we want them to perform a particular action.

This can potentially enable us to explore hundreds or thousands of ad possibilities within seconds, rapidly identifying and executing the biggest positive outlier. In short, we could peek into the future without actually running any live tests.