How to Product-Market Validate Your Startup/Product with Multivariate Testing

Is it possible to ‘simulate’ a product or business idea that only exists in your head right now?

What if we could understand exactly how customers would react to a hypothetical product — even down to the nitty gritty details of what your CAC (customer acquisition cost) would be, your ROAS (return on ad spend), and even which ad would work best for any given audience?

If this was possible, wouldn’t it allow us to rapidly asses product-market fit without even investing the ~$50–100k required to design, manufacture and ship a product? Yes, it would. It would be kind of like a cheat code, letting you take a peek into the future.

About a year ago, my team and I started to work with several well-capitalized entrepreneurs who were launching many different brands and products simultaneously — a business model referred to as a ‘venture studio’, which is when a single entity both founds and funds their own venture ambitions. This is in stark contrast to the ‘venture capital’ model, where one entity founds the idea and the other entity funds the idea.

Entrepreneurs running venture studios have a ton of advantages that skew the odds heavily in their favor — they have experience launching successful products before, they have capital, they have a reliable network of tested vendor relationships, and they have proven instinct.

Their biggest challenge actually has nothing to do with the typical challenges most founders face (funding, team building, ops), but instead has to do entirely with time. They have a lot of ideas, and not enough time to execute against all of them. Execution against a product idea is at least a 6–9 month process that requires product design, manufacturing/logistics negotiations, creative/web/content production, and launch campaign planning/deployment. All this, with only a 20% chance of successfully reaching product-market fit.

A holy grail for entrepreneurs is to find a way to skip all the time & work in between ‘product idea’ and ‘product-market fit.’

This is where we started playing with the idea of combining a design practice called Fake Doors and a marketing practice called Multivariate Testing (MVT). After much trial & error, we discovered that by combining the two principles in a strict methodology, we could successfully ‘simulate’ hundreds of product launch scenarios and tell entrepreneurs which products were worth launching and how (before any product even exists).

I’m going to peel back the curtain and show you how we do it for venture studios, how it works in a 3 step process, and how you can apply it to your product idea no matter who you are.

Step 1: Create a Multivariate Test

When I say ‘multivariate testing,’ almost everybody responds “Oh, that’s like A/B testing but on steroids.” In reality, the two are completely different practices that have almost nothing in common.

In an A/B test, you compare different ads that are run under the same circumstances. In a multivariate test, you identify variables (background image, copy, etc.), create multiple options (aka variants) for each variable group, and then you test every permutation (or, every possible combination of variants). This allows you to isolate and measure the impact of individual creative elements, empowering you to understand why your best ads work well (rather than just knowing which ad is the best one).

Multivariate tests are built in grids, as demonstrated above. They can range in size from only a few ads (the smallest possible grid would be 4 ads), or they can be hundreds/thousands.

Let’s say you’re launching a new juice product — there are a lot of variables that we all know would impact the success of this product (like flavors, packaging, color, branding, messaging, fonts, etc.). If we have a designer mock up these elements and combine them all together, we might get something that looks like this:

Above, you see a rendering of a hypothetical juice product, where we create an ad that represents every permutation of elements like colors, backgrounds, messaging, and layout. Below is another example for a dress company.

By doing this, we can ‘simulate’ hundreds of product-market fit scenarios at the top-of-funnel, allowing us to understand exactly which combination of elements will be the one that the market demands the most for our hypothetical product. This is critical because creative influences over 50% of sales outcomes in advertising, so it’s essential we use data to guide our creative instead of guessing or tastemaking.

Building a multivariate test can be a lot of work depending on how fancy you want to get . The bigger the multivariate test, the better your results will be — luckily, Marpipe can help you automate this entire process.

Step 2: Set Up a Fake Door

Fake doors is a very effective and common practice in the UX/UI community — some perceive it as evil or deceptive, but that all depends on how you go about it.

Setting up a fake door is a 4 step process:

1. Create a simple landing page that replicates a purchase or sign-up page. It should describe your product, complete with price, and appear to give visitors the ability to purchase it.
2. Drive traffic to this page from your multivariate test ads.
3. Users will be prompted to perform a conversion activity (aka click ‘Sign Up’, ‘Add to Cart’, or ‘Purchase’).
4. Users who click the conversion button should be taken to a page that says something along the lines of “Thanks for your interest! We actually haven’t launched (product name) yet, but when we do, you will be the first to know!”

This allows you to understand exactly what your conversion rate would look like without actually setting up the infrastructure to produce and ship a product. Major enterprises do this all the time — you’ve probably clicked something on Facebook before and got hit with this little gem:

Step 3: Component Analysis

After building your Multivariate Test and Fake Door, follow these steps:

1. Upload all ads from the MVT into an ad network of your choice (I highly recommend Facebook Business Manager) and be sure to tag each ad with the names of the unique component combination it represents.
2. Run each of the ad variants to about 1,000–1,200 reach and drive all traffic to the fake door. I recommend to run this test in Facebook Business Manager and to space out the campaign to run over 5–7 days.

Now, you have created your own data.

Warning: you can typically expect a lot of noise in this MVT data (aka, a lot of ads that did nothing or performed just about the same). The goal is to identify the positive outliers — which of them were far above the pack? The more ads included in your MVT, the more likely you are to generate more positive outliers.

Here are the questions your analysis should answer:

1. Which ads performed the best in the multivariate test? Alternatively, which group performed the worst? Important: we measure performance here by how many ‘conversion actions’ were generated by each ad. In this exercise, metrics related to clicks/engagement are vanity metrics and should be disregarded. Outcome: from this, we can estimate what our CAC and ROAS would be for our hypothetical product.
2. Which creative variables did those ads have in common the most? Outcome: from this, we can understand exactly how to market our hypothetical product the best way possible. If we tested different versions of our product, we can also tell which flavors/skews/colors/packaging is the best.
3. How well did our fake door convert? This is measured by the percentage of visitors who clicked the conversion button. Outcome: from this, we can understand whether we nailed it on price. If people bounced from the landing page at a high rate, we know that there’s interest in the product, but either our price or our landing page design was off.

With this system, we’ve been able to simulate hundreds of product possibilities for entrepreneurs and tell them exactly which products were worth launching and which ideas to scrap. Happy testing!