Back
All Articles

Does Creative Matter? The Meta-Experiment

AKA How to Run a Multivariate Test for under $500
Brett Friedman

About a year ago we had the idea for an experiment. A meta-experiment. A meta-meta-experiment.

We wanted to learn two things:

  • How much tiny details in ads make a difference on performance
  • How fellow marketers respond to getting ads tested on them
The experiment was a great success


Here are the results:

Tiny Details Have Massive Impacts on Performance!

For image ads: The best ad outperformed the worst ad by: 10.33x Clicks, +548.71% CTR

Largest performance improvements from a single variable change:

  • Text Layout: Center Middle to Bottom Middle = 5.16x Clicks, +315% CTR
  • Font Type: Sans-Serif to Serif = 5.81x Clicks, +394% CTR
  • Font Color: White-on-Black to Black-on-White = 2.74x Clicks, +201% CTR

For video ads: The best ad outperformed the worst ad by: 4.09x Clicks, +314.95% CTR

Largest performance improvements from a single change:

  • Intro Scene: Solo Human to Kitten = 3.46x Clicks, +201% CTR
  • Format: Static to Constant Headline = 2.75x Clicks, +215.83% CTR

Fellow Marketers Loved the Experiments:

  • For just $488 across 92 ads, we earned over 31000 impressions and over 2800 clicks

Comments fell into 3 (positive) categories:

1. Cool

  • "Very very cool."
  • "Cool little experiment!"
  • "Looked into this. Really cool."
  • "This is a really cool experiment 🔥🔥🔥"
  • "This is a really really freaking cool idea"
  • "This is really cool. Thanks for sharing the results! Is there a way to get in on any of your other studies?"

2. Meming

  • "I hypothetically just purchased this product and may or may not be enjoying it right now"
  • "hypothetically speaking, possibly. this isn't even a comment"
  • “Hypothetically speaking, this is a meme."

3. Conjecture on why it worked

  • 'It was the weird bottle’
  • 'The product’s not real, but if it was I wouldn’t have stopped'
  • 'The word experiment made me stop'
  • 'Ooh shiny thing and/or Helvetica Font'
  • 'FB glitched...'

To see the actual ads and performance data for yourself, check it out here: http://www.doescreativematter.com/archive/1

To comment and conjecture on the data yourself, tweet it out!

If you tweet, you’ll receive a special packet of data relevant to your industry 😉

Read our full creative testing guide

How to Run a Multivariate Test
for ad creative

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.