Back

Structuring Your Multivariate Experiment

Costs, KPIs, & Creative Variants
Marpipe

It's important to outline the structure of your upcoming multivariate creative experiment before building the actual creative. A good experimental structure will assure you find valuable results, including either positive and/or negative outliers. A poor experimental structure can result in unclear data which leads to confusion and a feeling of wasted effort.

01 Components of an MVT

When planning a multivariate experiment, there are a few key structural components that you need to decide on. These components include your KPI metric, avg. CPA for that metric, budget, timeline, and number of variables. Use the components that you are sure of (such as budget, KPI, and CPA) to help decide on the other components. In general, the KPI, avg. CPA, and duration are usually known or quick to figure out, while variables and budget need to be figured out.

In our July 2020 Does Creative Matter experiment we followed the below structure.

With a total of 27 ads (3 logo x 3 font x 3 background), our budget per ad was $9.25 and our daily budget per ad was $1.32. This experiment resulted in 3,464 total clicks with an average of 123 clicks per ad and helped us discover what creative components are most important for our campaigns.

02 Example Experiment Structures

Getting a large sum of results is easy in click-optimized campaigns, but multivariate experimentation is also very important for conversion-optimization. The biggest difference comes in that the avg. CPA is generally much higher for conversions (purchases, leads, etc.) than for clicks. We've outlined some examples of different structures to help you understand multivariate experimentation for your brand.

Product: Inexpensive Phone Cases

  • Product Price: $15 - $30
  • Components Known: Budget, CPA, KPI, Duration
  • Components Unknown: Variables

It's best practice to spend at least 80% of your avg. CPA on each ad in your experiment for good data collection - meaning we should spend about $10 per ad here ($12 CPA * 80%). Using the known components and our spend per ad, we're able to outline that we need about 350 ads for this experiment ($3500 budget / $10 per ad). With that information, we can decide on the variable groups and number of variants for those groups - 5 texts, 35 product images, and 2 designs.

Product: High-Quality Tactical Backpacks

  • Product Price: $55 - $98
  • Components Known: CPA, KPI, Variables
  • Components Unknown: Duration, Budget

With an avg. CPA for purchases of about $28 and the creative variables already chosen, all we need to do is figure out our budget and duration. Following the 80% of CPA rule, we want to spend at least $22 per ad here. Using that, we can figure out our budget of about $1,200 ($22 per ad * 54 ads). Lastly, we simply chose a duration of 14 days.

Product: Financial Services Application

  • Product Price: Free
  • Components Known: CPA, KPI, Duration
  • Components Unknown: Variables, Budget

For products with a very inexpensive CPA the best practice is to spend around $10 per ad. When it comes to things like App Installs or Clicks, a single result does not carry as much value as that of a singe result in purchase campaigns. Therefore, it's important to try and maximize results for these types of campaigns.

In this example, the CPA, KPI, and duration were all decided on initially, but the Variables & Budget were not. We could either simply set a budget first, or begin choosing variables and then find our budget from there. We decided to test 3 texts and 5 images for a total of 15 ads, which brought our budget to $150 ($10 per ad x 15 ads).

Product: B2B SaaS Platform

  • Product Price: $200+
  • Components Known: CPA, KPI, Duration, Budget
  • Components Unknown: Variables

Companies advertising products with a high CPA have two options for their experiments: spend more or run less creative variants per test. In this example, we have a preset budget of $2,000 and know our CPA, KPI, and duration. With a $70 CPA we should be spending at least $56 per ad ($70 CPA * 80%). From there, we know that we should run about 36 ads ($2,000 budget / $56 per ad) - so we outlined 3 text, 4 graphic, and 3 design variants.

Product: Innovative Tech-Enabled Mattress

  • Product Price: $1,500
  • Components Known: CPA, KPI, Duration, Variables
  • Components Unknown: Budget

High-ticket products shouldn't expect the best practices of multivariate experimentation to be any different for them, they'll just need to spend a large amount per ad to capture significant results. In this example, we wanted to test 18 ad variants (3 image * 6 text) and needed to spend about $720 per ad ($900 CPA * 80%). That experimental outline means we need a budget of $12,960 ($720 per ad * 18 ads).

Product: Nationwide Media/News Outlet

  • Product Price: Free
  • Components Known: CPA, KPI, Duration, Budget
  • Components Unknown: Variables

For blogs, news, and similar types of media sources that are optimizing their advertisements for clicks or other low-cost results (reach, engagements, etc.), it's just as important to build a data-driven creative process as it is for any other brand. The main difference between these types of optimizations is that the CPA is often very inexpensive.

In this example, we have a CPA that is too low to use for our spend per ad, so we need to default to spending about $10 per ad. With a budget of $5,000, we can run 500 ads at $10 per ad ($5000 budget / 500 ads).

03 Summary

The rules and practices used throughout the examples above can be adjusted for every product with any goal. Just input the components that you have to decipher the components that you don't yet have. Here's a quick rules breakdown:

  • Components: KPI metric, avg. CPA for that metric, budget, timeline, and number of variables
  • Spend per ad should be at least 80% of your avg. CPA
  • With low cost KPIs (avg. CPA below $10), it is best to spend about $10 per ad
  • When choosing your duration, it is best to run experiments in 7-day increments (7, 14, 21, 28 days), to capture each day of the week evenly.

Read our full creative testing guide

SIGN UP FOR THE EXPERIMENTAL METHOD NEWSLETTER
TO GET THE GUIDE

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.