Do you really need a control group?

A control group are the marketers’ Saint Graal. Or so it seems. A majority of articles found online encourage Control Group testing.

Why? Because this method of testing goes beyond classical attribution models and their setbacks.

It accurately measures the incremental revenue a marketing strategy can generate. It does sound good, looks rather easy.

But there are some drawbacks to the method that I want to help you identify.

What are control groups?

Let’s start from the beginning: what is a control group?

A control group is a part of your database that you willingly decide to withhold from your marketing strategy. Your test group on the contrary will be exposed to your strategy.

In practice, how to proceed?

You simply have to split your customer database into two groups, a control group and a test group.

You then should expose your test group to the strategy you want to test, making sure your control group is unexposed.

Then, simply measure the incremental revenue generated by your new strategy on the test group.

That’s all folks. Easy indeed.

Unfortunately, reality looks a little different.

The hardest part about control groups is not about how you set it up.

The challenge comes when it’s time to analyze your results, and how you make sure your results are significant.

If you base your analysis on falsy results, you are risking an inaccurate reading of your test results, which could then lead to making the wrong decision.

To the point of causing a decline in the performance of your entire CRM strategy.

No one wants that.

To help you get strong results on your future control groups, we’ve grouped our best advice on how to get significant results.

They are based on two things:

1. Best practices about the control group methodology, used in many different fields, like in medicine for example, where control groups are used to validate a treatment – or a vaccine against Covid for that matter.
2. Our own corporate experience, focused on marketing, with insights from Nathan Desdouits, Data Product Manager, who holds a PhD in biology and has managed control groups for both CRM campaigns and scientific research. He wrote a blog article on Control Groups if you’re curious.

Control groups are not the right answer to each situation

Control groups cannot be set up in the snap of a finger.

For this reason, you should really start by asking yourself if a Control Group test is the right solution to prove the point you need to make.

You might be facing two different situations:

• Situation 1: you need to compare two strategies. For instance, a targeting method like Splio against an in-house targeting method. You want to know if one method is more efficient than the other.
• Situation 2: you consider changing your marketing strategy and you’re wondering if it is the right decision for your business. For instance, you’re thinking of adding targeted campaigns to your current campaign plan. Will this help increase revenue or just fatigue customers?

Situation 1

If you’re in situation 1, you have a time-sensitive issue at hand. You need to know fast if you should change your targeting tool, and if you’d better change your current solution. In that case, a control group might not be the best solution as it takes time to get results. A simple series of A/B tests would be enough and will help you know which method is more effective for your needs.

Situation 2

If you’re in situation 2, you are looking to address a deeper issue. Indeed, it implies changing your marketing strategy completely. In this specific case, you need absolute scientific evidence and precision that what you’re considering is right. Therefore, you should use a control group to decide.

In any case, time is key and should help you decide.

Do you want to get results fast to make your decision, or do you need to be absolutely and irrevocably sure about a change you are about to make?

A control group will not always be the best answer to all your day-to-day questions, but with some time it can provide conclusive evidence to help you confidently adjust your strategy.

What you need for a control group to work

Now you know whether a control group is the right testing protocol for you.

If you find out it is, you still have some challenges ahead.

I’ve put together some tips for you to get set up with the best conditions possible, and get fast results.

You need to define a clear goal to your test

Depending on your needs, you will need to set a single goal that will reflect the purpose of your test.

If the purpose of your control group test is to help you prove that your new strategy is driving more revenue, then you will want to measure the incremental revenue generated by your new method.

Measuring this goal will be your lighthouse in the storm, the main focus of your control group test.

You need a drastic change to see significant results

If you decide for a control group to confirm whether your change of strategy is the right one, you will need to make major changes to your strategy to see an impact in your test results.

The strategy you are experimenting should go all the way.

Let’s take an example.

If the purpose of your test is to prove that your Tinyclues campaigns have an impact on revenue, you should compare the results of your marketing plan including Splio campaigns against the results of your marketing plan excluding Tinyclues campaigns.

The difference between your control group and your test group should therefore only be the addition of Splio campaigns to your planning.

The effect of a single campaign will be limited. Therefore, you will need to run your test on several campaigns in order to observe an impact: the more campaigns you will be sending to your test group, the more likely you will get accurate results, and the more revenue you will generate.

Size does matter

I have a simple equation for you: the more data you get, the more confident you can be in your results.

A large control group will allow the collection of more data, which will decrease uncertainty and help you measure incremental revenue faster. But it also means that you will be freezing a large share of your database, and that you will be missing out on revenue.

On the contrary, a small control group means that you will be collecting less data than with a large control group. You will need more time to get significant results. But it’s a safer bet as you will lose less revenue in the process.

So if your issue is time-sensitive, the bigger the control group, the faster you will get significant results to make a decision with.

Another important question is whether your database is large enough to support a control group.

Indeed, there is a difference between a database with 3 million or 100,000 active customers. That difference will have a direct impact on the size of the control group that you will be able to make.

It will not take you the same amount of time to get significant results for a control group with 50,000 users compared to a group of 1.5 million users.

Patience is key

If you follow our advice and define a large control group with a significant change of strategy, you might get results fast.

Fast for sure, but not that fast.

Retaining a fixed control group over the course of several months will allow you to cumulate effects of multiple campaigns, and that is exactly what you are looking for.

So, control group or not?

As we have seen, control groups, though appealing, do not fit every single situation.

You will need to evaluate your needs, the time pressure you have on your test, in order to define what’s best for you.

There are a few best practices you should follow to get significant results with a control group on your marketing campaigns.

Always remember:

• Clearly define what you want to demonstrate: I want to measure the impact of this tool on my CRM revenues.
• Compare two very different strategies for significant results: your test group will get CRM full base campaigns, while your control group will be frozen and will not get any campaigns.
• Size and patience does matter: it’s ok to freeze 10% of your database for a year, if it’s necessary to get significant results.

In my next article I will take you through how to handle the statistical significance of control groups, with some real life examples at hand.

Stay tuned!