How Much Does it Cost to Run a Multivariate Test?

The most frequently asked question about multivariate testing is, "How much does it cost?". There's a belief that running hundreds of ads will cost millions of dollars and, logically, what follows is an assumption that what you learn from MVT is not worth what you put in. However, to most people's surprise, MVT can be used regularly by advertisers and still remain relatively inexpensive; you just need to understand how to structure your experiments and what results you can expect from them.

Understanding MVT experiments is not as difficult as it might sound, but to build and run your own correctly you need to be diligent in your creative and campaign set ups. The goal is to both find incredible performing ads and to learn what isolated creative components your audience likes best (colors, copy, images, graphics, etc.).

To get started, you'l need to calculate how much ad spend you require for your first MVT. Finding that number depends on your average cost per result (based on your KPI) and the number of ads you'd like to run.

Calculating Cost Per Result (CPR)

Finding your CPR should be quick and easy if you already have past advertising data, you may even already know that number in your head! However, if you're unsure of what CPR to base your experiment set up around, look at the average CPR of any campaigns you've run with the same KPI and go with that. It's important to note that your cost per result depends on what 'result' you're after. KPIs lower down your funnel cost more money to acquire i.e. purchases cost more than leads cost more than clicks.

If you do not have any past advertising data, this may be a bit more difficult. In that situation, you should look up the average CPR for your KPI within your industry. That will at least give you an estimated CPR idea to build your experiment off of.

For this example, we'll assume you're taking the CPR for your last purchase campaign - $20 per purchase.

Calculating Number of Ads

Now you need to figure out how many ads you're going to make. Continuing on this example, we've outlined three creative variables for your experiment:

1. Creative Design
2. Image
3. Messaging

Creative Design simply means high-level layout; the way your images, texts, graphics, logos, etc. are displayed in the ad creative. It's often best to use your top performing ad creative as a framework for your designs, this will help you see how your best ad can be improved by using other creative assets! Once you've organized your designs, swap in & out the images and messages to build your multivariate matrix! The important thing here is to make sure every design, image, and message variant is combined with every other design, image, and message variant.

After building out every version of your creative, you have a total of 50 ads! Here is the breakdown:

1. Creative Designs - 2
2. Images - 5
3. Messages - 5

With 2 designs, 5 images, and 5 messages, you've made 50 ads (2 x 5 x 5) that are ready for testing. To run those ads against a single audience at a $20 CPR, you'll need a budget of $1,000 (50 ads x $20 per ad). If that's a bit too expensive, we suggest spending at least 75% of your CPR on each ad, which would make the total ad spend for this test $750 (50 ads x $15 per ad).

Experiment Results

Now that you have a basic understanding of building out your test structure, you can use that to understand your expected results! If you were to run the $1,000 campaign, you could expect around 50 purchases (assuming CPR is accurate). If you chose the $750 experiment, the number of purchases should be about 37.

Ideally, you'll want to make sure your experiment gets at least 30 purchases for the data to be relevant. However, even a smaller number of purchases can give you compelling data if there are strong positive or negative outliers.