Back
All Resources

A History of Multivariate Testing

The Development of MVT Techniques in Marketing
Brett Friedman

From its early statistical beginnings to its complex and dynamic state of being that we know today, multivariate analysis has changed the way we connect to our world and pushed humanity forward in countless ways. Understanding its past and its evolution to its modern state can help businesses use it to their advantage and make better decisions.

The Multivariate Timeline.

In order to understand multivariate analysis and use it correctly, you must learn its history.

  1. A Brief History of Multivariate Analysis in Statistics and Math
  2. A Brief History of Marketing, Market Research, and the Development of Ad Tests: Precursors to Multivariate Testing
  3. The Intersection of Math and Marketing: Multivariate MarTech

Soon you'll return to your coworkers a gentle-human and a scholar, fully educated on the combination of art and science in favor of better business decisions.

A Brief History of Multivariate Analysis in Statistics and Math

1776 is independence year for America. More importantly, it's the year Joey-Lou Lagrange derived the first multivariate technique while studying statistical means. In all it's beauty and grace, it was called "The Multivariate Normal Distribution". As it sounds, it's an extension of your average, every day Normal Distribution. The one you know and love with her bell curves and occasional long tails.

The OG uni-variate normal distribution and the bi-variate normal distribution. Multi-variate (not pictured) includes at least one more dimension

1827, Laplace adds multiple linear regressions to the mix, first allowing us to isolate the effects of a single variable on an entire function e.g. the flavor of a veggie burger.

Flavor = Patty + Tomato + Avocado + Grilled Mushrooms + Un-Toasted Bun

Multiple variables. Each one pulls their own weight towards your enjoyment (or disgust) of it. To find out how much each component adds to your experience of its flavor, you can use Laplace's multiple linear regression solutions.

One young life later, it's 1846 and Bravais is drawing maps in 3 dimensions. Ground-breaking. Galton takes note, but not before Gustav Fechner applies the multivariate mindset to psychology in his 1860 treatise on Psychophysics: an olive-branch discipline measuring interactions between physical phenomena and psychological states.

How does this make you feel?

Fechner was kind enough to discover the just-noticeable difference, an average minimum threshold for a person to sense or perceive a physical stimulus. Spoiler: it's a 10% change. That discovery would evolve into Zajonc's Mere-Exposure Effect, a likely culprit for why annoying salespeople and pop-up ads bombard you: 'aided recall'. More on that later. For now, know that Fechner developed 'Multi-factor' experiments to discover these things, an effective off-shoot of multiple linear regressions in which:

"Everything that could be varied was varied; everything that could be measured was measured; everything that could be recorded was recorded." - Gustav Fechner

Back to Galton. 1863: he's cold-callng weather stations for juicy meteorological data; wind direction, temperature, barometric pressure. He uncovers "Anticyclones".

Count the variables:

  1. Wind Direction – Univariate
  2. Temperature – Bivariate
  3. Barometric Pressure – Multivariate

Galton's running multivariate models on geographical maps of weather to find patterns.

Galton's wrath: The official mark of humanity's first exposure to visual multivariate reality

Another 19 years pass and Galton tries crossing height, artistic ability, and chances of getting tuberculosis. I'm sure he found something very special.

In the meantime, Adolphe Quetelet, a Belgian Astronomer, uses statistical constants for Nature's constants boiling chaotic causal combinations into single cause-effect relationships. Oh and he messed around with "The Average Man", someone with average (mean) characteristics (e.g. 1.95 arms, legs, hands, eyes, and so on).

An above-average man

Another Francis, Francis Edgeworth, took note from Quetelet's above-average features and invented correlated averages to take Francis Galton's lead on Biological applications. He claimed multivariate measurements were normalized because each organ of any organism was itself normally distributed across individual observations.

That of course, is absurd, as Karl Pearson was apt to call out when he solved the 36 dimensional problem which Edgeworth called impossible, using multi-dimensional goodness of fit models.

That was all between 1892 and 1895. Towards 1907, Udny Yule introduced partial regressions and multiple correlation coefficients. Returning to the vegan burger:

Flavor = (x)Patty + (z)Tomato + (α)Avocado + (β)Grilled Mushrooms + (δ)Un-toasted Bun

Multivariate Meat

Those roman and greek letters represent weights (a.k.a. correlation coefficients). Previously, all variables were held constant to find any one number. Now, multiple weights and multiple variables can change at the same time and you can still learn the values of all the rest. This is the true basis of multivariate analysis.

Thanks to Yule, from 1920 onward, R.A. Fisher took off developing multivariate analysis along with a major portion of Inferential Statistics.

100 years later, we've got MANOVA, PCA, Factor Analysis, Correspondence analysis and more. It's all about data. Big data. Until now, we had thousands of data points as global maximums. With globalization and increasing amounts of data-tracking tech in all fields and forms, we have millions of multi-dimensional data points to parse. Far more than we can handle. So mathematicians and statisticians focus on boiling that information into meaningful, predictable patterns amidst the chaos.

A Brief History of Marketing, Market Research, and the Development of Ad Tests: Precursors to Multivariate Testing

Move your mind back a century and restart your life in 1900. The automobile, the airplane, the radio: all novelties. Marketing at this time was nothing more than supply chain and distribution logistics embedded in deep economics.

In 1910, Marketing gets a promotion to include advertising and selling.

1915: Ralph Starr Butler and the American Marketing Association incorporate Census Bureau survey design and sampling into their systems essentially coining "Market Research". An event that coincides with Fisher's contributions to statistics. And radio advertising.

In the 20's a man named Claude Hopkins comes up with "Scientific Advertising", planting seeds and tending roots for advertisement testing.

"Almost any question can be answered, cheaply, quickly and finally, by a test campaign”…"Now we let the thousands decide what the millions will do. We make a small venture, and watch cost and result. When we learn what a thousand customers cost, we know almost exactly what a million will cost. When we learn what they buy, we know what a million will buy... Then we try test campaigns to try out new methods on advertising already successful. Thus we constantly seek for better methods, without interrupting plans already proved out." - Claude Hopkins

It's all there. The sentiment of test and learn, learn and grow. Back in the 20's.

In 1932, John Caples takes Hopkins roots and fleshes out a tree. He tests appeals, headlines, illustrations, layouts, colors, text format, messaging, offers, and even mediums.

Of course it was in the 40's when attribution got good enough to get true results through direct-response mail. Robert Collier was one of the first to incorporate Psychology into his ads essentially making 1940's clickbait thanks to the closed feedback loop direct mail provided. He's also one of the first to use and discuss statistics in advertising (e.g. open rates, conversion numbers, etc.)

Robert Collier invented clickbait

20 years later, Ford Foundation and Carnegie Corporation catch on and begin publishing reports: multi-disciplinary, quantitative reports measuring psychological concepts such as 'aided recall', lifestyles, self-concepts, social classes, frames of reference, and every other major piece of research. These were used with the world's first focus groups to create 'decision sciences'. Oh, and telemarketing takes the stage.

As Vietnam War protests raged on through the 60's, marketers needed new and better techniques. Behavioral scientists and mathematical modelers were brought in to conduct "Motivational Research". What makes people do things? What makes people buy things? Question we return to day in and day out to this very day.

Motivational

The late 60's highlighted management and the "Marketing Mix" which included the 4 P's: Product, Place, Price, and Promotion. Qualitative data takes the limelight and reports focus on determining effectiveness through attribution.

Guys like David Ogilvy and George Lois start showing up emphasizing ad creative, creative testing, and "Big Ideas". They called it "The Creative Revolution". Positioning and brand image become tantamount to success for a company. Marketing becomes a major accounting line item for the first time.

“Never stop testing, and your advertising will never stop improving.” - David Ogilvy

By 1970 we had a whole new subfield called Consumer Behavior, legitimized by the computer's mass production, invention of statistical softwares, and publishings of Journal of Consumer Research. People also figured out how to sell things online. They called it: e-Commerce. Clever. This surprisingly slowly led to internet marketing.

The 80's invented spam and guerrilla tactics, but it wasn't until the 90's that web marketing took off. With the Dot-Com boom, thousands started online companies expecting to make millions. Marketing exploded into all different directions: display ads, email marketing drips, search ads, interstitials, content, CPC, and so on.

You know because you were there.

The Intersection of Math and Marketing: Multivariate MarTech

From then on, attribution became more direct and research became more legitimate. The once loose connection between ads and purchases became significantly more exact. Someone clicks your ad, goes to your site, purchases your product. A single path. With more accurate attribution comes more meaningful research. Ads become survey items, purchases become votes. With more data, more patterns are available to view, more predictions can be made, and more tech can be developed to augment this previously manual process.

Over the past 20 years, approximately 7000 of these softwares have been developed to collect, parse, and exploit that data.

7000. 7000 softwares. 7040 to be exact.

To stay on point, we'll focus on those few softwares pushing boundaries of ad testing: multivariate martech. Multivariate martech, operationally defined, is any software that tests a marketing channel by changing many components at once, in order to find how each individual component affects the outcome (conversions of some kind).

I've selected five major MVT technologies as examples of what's possible. This is by no means a comprehensive list, but it includes varied applications of MVT to advertising.

  1. Marpipe: MVT for Ad Creative
  2. Optimizely: MVT for Websites & Landing Pages
  3. Mailchimp: MVT for Email
  4. Lightning AI: MVT for Targeting/Audience
  5. Beeswax: MVT for Programmatic Bidding

Marpipe: MVT for Ad Creative

That's us. We use multivariate analysis to test ad creative: designs, imagery, messaging, anything you can put in ad, we can test. Video or image. Change all these variables at once to find how each individual variant affects your ad.

48 variants each with an average 6 possible states = 2.24^37 = ~∞

Optimizely: MVT for Websites & Landing Pages

The first and foremost practitioners of MVT for landing page optimization. They allow you to use dozens of different images, buttons, layouts, and links on your site to see which order makes your audience most likely to convert.

Courtesy of Invesp

Mailchimp: MVT for Email

Mailchimp is the sweetheart of email marketers everywhere. Recently they added basic mutivariate testing to their platform allowing users to run multiple subject lines, from names, and body content. Thanks to Mailchimp, you can learn like you're a direct-response mail marketer of the 40's.

Buy Me from Brett vs. Buy Me from Mr. Friedman vs. Use Mailchimp from Brett vs. Use Mailchimp from Mr. Friedman

LightningAI: MVT for Targeting

LightningAI uses AI to test 100s of audiences with your ads, isolating individual groups that respond best to your product or service. To find these audiences, they systematically vary targeting options such as interests and demographics i.e. they use MVT.

Millions of people, thousands of audiences, hundreds of targets

Beeswax: MVT for Programmatic Bidding

Beeswax is a relatively new (2015) ad publisher and the world's first BaaS, Bidder-as-a-Service. Beeswax allows multivariate bid models for your programmatic strategy.

Click the pic to learn more

Quick Recap & Conclusion

From reading this article, you got:

  1. Historical stories to tell your friends at parties
  2. Stronger conceptual understanding of quantitative marketing
  3. A good experience and a positive mood... and presumably years of upward spiraling positivity as you reap the continuous rewards of having deep knowledge on a concept that will become increasingly relevant over the next century as data continues to drive marketing performance.

If you know any companies using MVT that are well differentiated from what I mentioned here, please reach out to me: brett@marpipe.com

I'd like to thank Dr. Joseph Hair and Dr. Barry Babin, the foremost scholars on multivariate analysis in marketing, for sending me information including their article entitled "History of Multivariate Education" from which I drew greatly.

Read our full creative testing guide

SIGN UP FOR THE EXPERIMENTAL METHOD NEWSLETTER
TO GET THE GUIDE

Download Now
View Sources

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
"This is a block quote"
  1. This
  2. is
  3. an
  4. ordered
  5. list

This text is bold

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to Run a Multivariate Test

The Beginner's Guide

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Tiffany Johnson Headshot

How to Run a Multivariate Test
The Beginner's Guide

Plus, Get our Weekly
Experimentation newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.