Creative testing
budget calculator

Want to maximize your chances of finding winning ads and creative assets with high confidence? Use this calculator to see how much you should spend.

A/B VS. Multivariate Testing

This calculator will work for either A/B or Multivariate testing structure.

If you’re currently only  A/B testing, enter the total number of ad variants you plan to test.

If you’re multivariate testing (bravo, BTW), just enter the number of variants in your largest variant group.

Ex. You are testing 3 headlines, 4 background images, and 2 layouts. There are 24 total variants from 3 separate groups. Since the background variant group is the largest, you would input 4 into MAX VARIANTS.

Action Rate

The more actions (clicks, purchase, leads, etc.) attributed to your ads, the cheaper it will be to gain high confidence on your results. To estimate budget, we need to understand how many actions we need to collect for a given number of impressions.

To estimate your action rate, look back at your historical ad performance and divide the number of total results for your desired action by the total number of impressions it took to obtain those results.

Action rate = Total actions/Total impressions

If the product or service being advertised is expensive, it may take more impressions to reach high confidence for an action like purchase. In these scenarios, it may be necessary to target a metric like clicks or engagement to gain high confidence.


Sometimes, meaningful differences in results exist but are small, and difficult to detect. Detectability lets you decide how small of differences to find. The lower the detectability, the more budget you will need to reach high confidence and the more likely you are to find a meaningful difference between your variants.

You could spend thousands of dollars and still not reach high confidence on anything being tested. Why? Because there’s nothing to be confident in. Statistical significance is the likelihood that the difference in conversion rates between a given variation and the baseline is not due to random chance.
If there is no meaningful difference between what is being tested, then we most likely need to change what we are testing. It’s entirely possible that the changes between the test variants just don’t matter.

Statistical Significance and Confidence

Imagine we want to predict how often a pro basketball player is going to make their free throws.
If they take 10 shots and make all of them, their conversion rate is 100%. But is it reasonable for us to predict that they will continue to make 100% of their shots going forward? Probably not.
Luckily, there is a metric that helps us make predictions given what we already know.

Confidence is another way to describe probability. For example, a 95% confidence level means you are confident that 95 out of 100 times the estimate will fall between the upper and lower values specified by the confidence interval. The confidence interval is simply a range between two numbers that we feel we can accurately make a prediction within.

Based on what we know of our pro basketball player, we can predict that they will make anywhere between 1 and 9 of the next 10 shots. Not very helpful, right? Well, we need to collect more data. The more data we collect, the more confidence we gain and the smaller (and more accurate!) our confidence interval becomes.

Let’s say after collecting over 1,000 observations of our basketball player, we can now predict with high confidence that they will make anywhere between 7.4 and 8.1 of the next 10 shots.

The more we observe the better we become at predicting the future. And here, the same goes for your ad creative.
If MVT: The largest number of variants within a variant group in your test.
If A/B: The total number of variants
Input your average CPM. This way we can estimate how much it will cost to get the impressions that we need for significant results.
Choose the type of KPI you are going to use to analyze your results.
KPI’s with higher action rates (like engagement) gain confidence more easily and have a higher likelihood of detecting a meaningful effect. You will gain high confidence ONLY if a large enough difference in performance actually exists between the variants you are testing.
Input your expected action rate. Depending on your goal the action will be different.
Ex. If your goal is clicks, input your average CTR
Ex. If your goal is purchases, input your average impression to purchase rate
- -
Your recommendation
By supplying my contact information, I authorize Marpipe to contact me with other communications about Marpipe’s products and services. See our Terms & Conditions for more details.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
SolutionsMultivariate Testing
InformationPricingRelease NotesBlogPressHelpSupportContact UsNewsletter Signup
CompanyAbout UsPrivacy PolicyTerms of Use
BlogHow to Write Compelling Ad Headlines for Creative TestingRelease NotesWhat a Losing Ad Tells Us About How to Make a WinnerNew Research Shows Marketers are Really Bad at Predicting Winning Ad CreativeWhy Multivariate Testing is the New Standard in AdvertisingMore
Follow Us