Beta
Well this is exciting! We're happy to have you on board.
Fill out your info and book a product tour.
No, I don't want to see how it works
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Does Creative Matter? The Meta-Experiment

AKA How to Run a Multivariate Test for under $500
Brett Friedman

About a year ago we had the idea for an experiment. A meta-experiment. A meta-meta-experiment. We wanted to learn two things:

1. How much tiny details in ads make a difference on performance
2. How fellow marketers respond to getting ads tested on them
The experiment was a great success


Here are the results:

For image ads: The best ad outperformed the worst ad by: 10.33x Clicks, +548.71% CTR

Largest performance improvements from a single variable change:

  • Text Layout: Using Center Middle text instead of Bottom Middle text leads to 5.16x more Clicks and a +315% increase in CTR
  • Font Type: Sans-Serif compared to Serif font leads to 5.81x more Clicks and a +394% increase in CTR
  • Font Color: Displaying White-on-Black as opposed to Black-on-White leads to 2.74x more Clicks and a 201% increase in CTR

For video ads: The best ad outperformed the worst ad by: 4.09x Clicks, +314.95% CTR

Largest performance improvements from a single change:

  • Intro Scene: Showing a Solo Human compared to a Kitten leads to 3.46x more Clicks and a +201% increase in CTR
  • Format: Keeping the headline Static as opposed to a Constant Headline leads to 2.75x more Clicks and a +215.83% increase in CTR

Fellow Marketers Loved the Experiments:

  • For just $488 across 92 ads, we earned over 31000 impressions and over 2800 clicks

Comments fell into 3 (positive) categories:

1. Cool

  • "Very very cool."
  • "Cool little experiment!"
  • "Looked into this. Really cool."
  • "This is a really cool experiment 🔥🔥🔥"
  • "This is a really really freaking cool idea"
  • "This is really cool. Thanks for sharing the results! Is there a way to get in on any of your other studies?"

2. Meming

  • "I hypothetically just purchased this product and may or may not be enjoying it right now"
  • "hypothetically speaking, possibly. this isn't even a comment"
  • “Hypothetically speaking, this is a meme."

3. Conjecture on why it worked

  • 'It was the weird bottle’
  • 'The product’s not real, but if it was I wouldn’t have stopped'
  • 'The word experiment made me stop'
  • 'Ooh shiny thing and/or Helvetica Font'
  • 'FB glitched...'

To see the actual ads and performance data for yourself, check it out here: http://www.doescreativematter.com/archive/1

To comment and conjecture on the data yourself, tweet it out!

If you tweet, you’ll receive a special packet of data relevant to your industry 😉

Get A Product Tour

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.