All marketers perform tests to discover which subject line gets the most email opens, what homepage content improves your search engine optimisation (SEO), what button colour leads to more purchases, and so forth.

Even though, now and again, you have to answer a different type of question:

“What can I do to cause an incremental improvement versus doing nothing at all?”

As a way of example, let’s say you want to run an email campaign with a 20% discount offer to customers who are likely to want to buy knitwear. In order to figure out whether or not this campaign is successful, you must know how many people made a purchase from this campaign that would not have done so if they never got the discount email.

Say if 80 people were to buy a new knit top, but almost all of those people would have done so anyway (although now they have got a nice 20% off coupon), then you’ve actually damaged your company by providing discounts where they were not required.

Although, by correctly-designing and executing an experiment with a control group (or holdout sample) you are able to predict with better precision whether or not the campaign was effective. You would need to take a random subset of the population and do nothing to them, and then look at what they do compare to the group that received the treatment.

Whilst it can be difficult to prioritise experiments, they can give you a much better idea of the true return on investment (ROI) for your marketing, which can be a big pay off in the long term.

These are 5 key ways to efficient marketing experiments:

1. Keep The Control And Test Groups Balanced.

It is crucial to stay disciplined about control groups – the temptation is generally to reduce the control group down to maximise the earning potential of the test. However, the main issue is that without an adequately sized control group, deriving meaningful results and conclusions can become next to impossible since you don’t have a good read on baseline customer behaviour.

2. Ensure The Control And Test Groups Look The Same.

Your control group must be the same as the test group. It must be identical. If you are trying to see how a 10% discount works for mothers in West Sussex, then both your control and experimental groups must be mothers from West Sussex.

To do this, you’ll want to define your segment, and then randomly assign members to the test and control groups. Many email service providers can do this for you; otherwise you’ll need to use Excel or another tool to create the random groups yourself and then upload it accordingly.

3. Eliminate Other Variables

Because you are trying to see the impact of one idea, and only one idea, then you need to make sure that the test and control groups are otherwise identical in treatment. If one group receives a daily email about socks, then the other group needs to get that same sock-loving email; otherwise the results are tainted and therefore meaningless.

4. Keep It Simultaneous

It’s important to remember that things do change; strategies shift, seasons change, competitors rise and fall. Bearing in mind these factors of business, you need to ensure that your test and control groups are being tracked at the same time. You cannot compare last month’s holdout group to this month’s test group.

5. Make Certain That Results Are Trackable

The entire concept is to be able to measure the test and control groups against each other, so ensure you know what you are tracking and how you are going to track it. And since the control group is not getting any emails, you can’t use clicks or opens to compare. Customers cannot open an email they never received. Look at the revenue, orders, site visits, conversion rates, and so on during the test period to see whether the campaign did well enough to merit further investment.


To measure the complete impact of your next marketing campaign (and see its true incremental effect), make sure to have a holdout group that do not receive the campaign so that you can properly compare results.