3 Comments

Summary:

When it comes to marketing, many feel it’s more an art than a science. But by setting aside a control group you can turn marketing campaign results into measurable data.

coupons1

You’re spending plenty of time and money on your customer retention campaigns, but are you effectively measuring the effectiveness of your campaigns – in monetary terms – in order to optimize future campaigns and maximize the revenues they generate? That’s why you must introduce a scientific approach to your customer marketing efforts.

Instead of focusing on email open rates and click rates as the primary means of measuring campaign effectiveness, it is critical to focus on the monetary uplift generated by each campaign. The way to do this is to treat every campaign as a “marketing experiment.”

Control groups: The basis of a scientific approach

controlThe key to determining the true effectiveness of any customer marketing campaign is the proper use of a control group. A control group is a subset of the customers you’re targeting with a particular campaign who you decide will not receive the campaign.

The members of the control group are randomly selected to represent the entire target group of customers. In other words, they should be similar to the members of the entire group and thus be exposed to the same set of conditions, except for the particular marketing campaign being tested.

For example, let’s say that an online retail marketer has performed some customer segmentation to select 10,000 customers to receive a particular offer. He will send this campaign to only 9,500 of them (the “test group”), setting aside a randomly-selected subgroup of 500 customers (the “control group”) who will not receive it. Once the campaign is over, the marketer will determine the effectiveness of the campaign by comparing the additional revenues generated by the test group with those generated by the control group.

Determining the ideal size of your control groups

It is very important that the control group is a representative sample of the overall campaign population. When the control group size is large enough, the process of random selection will always result in a control group that effectively represents the entire group.

The sample size you need depends on the size of the overall campaign population. For 10,000 customers, as in the example above, 5 percent is sufficient. As a rule of thumb, smaller campaigns require a larger percentage to generate a valid control group. So, for campaigns targeting less than a couple of thousand customers, it’s a good idea to use 10 percent to 20 percent instead.

There is one additional factor to consider when deciding upon the size of your control group: your expected response rate. When you expect a particularly low response rate for a particular campaign (for example, when sending an offer to long-dormant “churn” customers), you will need a larger control group in order to obtain statistically significant results. On the other hand, if you expect a particularly high response rate (for example, sending a bonus to your best customers), a smaller control group will be sufficient.

Analyzing the results of your marketing experiment

Let’s imagine that, before starting to use control groups, you sent a marketing campaign to all 1,000 of your best customers. You offered them a 10 percent discount on every product in your store for one full week.

The campaign’s metrics might have looked like this:

test1

At first glance, the campaign results look terrific! 20 percent is a high response rate, and 200 customers spending an average of $200 apparently generated an additional $40,000 of revenue in one week.

But since you didn’t run this campaign as a scientific experiment, you really have no way of knowing how many of these 1,000 customers would have made a purchase this week anyway, or how much of this $40,000 would have been spent by these customers even without the campaign.

One might think that by simply comparing the purchase rates and spend amounts of a similar group of customers from a previous time period (when no campaign was run) to the current period (in which the campaign ran) can reveal how much additional revenue was generated by the campaign. The reason that this comparison is not valid is that there are always numerous other factors affecting customer behavior from one period to another. It is crucial that the comparison between the test and control groups is made during the same time period.

Running the same campaign as a scientific marketing experiment

If you had run this campaign as a marketing experiment, using 10 percent of the target customers as a control group (who did not receive the campaign), the campaign’s results might have looked like this:

test2

Let’s take a look at the control group. Even without receiving any offer, 14 percent of your best customers made a purchase from your store during the week in question anyway — spending an average of $150 each. Since the control group represents the entire target group, we can extend the control group’s buying behavior to represent the scenario that no campaign had been run at all. Thus, we could have expected that, absent any campaign at all, the entire target group would have made purchases totaling $21,000 (1,000 customers x 14 percent x $150).

So the question we need to answer is: how much additional revenues resulted from the marketing campaign? (For the sake of simplicity, we are not considering any costs to the company of offering the 10 percent discount.)

In actuality, the set of all 1,000 of your best customers spent $38,100 this week: the 180 customers who received the campaign spent a total of $36,000 plus the 14 customers who didn’t receive the campaign spent an additional $2,100.

Thus, the actual gain generated by this marketing campaign is $17,100 ($38,100–$21,000), a far cry from the $40,000 apparent gain we concluded before using a control group.

Keeping your scientific experiment clean

To retain the integrity of the results, it is important to ensure that no additional factors under your control are influencing customer behavior – just as in any scientific experiment. In other words, during the measurement period of a particular campaign (usually a number days), the test and control groups should not be exposed to any other targeted offers or incentives. If customers are receiving multiple simultaneous campaigns, it becomes impossible to measure the effectiveness of any one campaign. In our experience, many marketers fail to realize the importance of isolating their marketing experiments.

In conclusion, treating your customer marketing campaigns as a scientific marketing experiments allows you to obtain an accurate understanding of how well your campaigns are working – in monetary terms.

Using control groups, you can determine the actual monetary uplift of every marketing campaign. By testing many campaigns, and keeping close tabs on the true effectiveness of each, you will be able to gradually optimize all your marketing campaigns for maximum financial results, as well as improved customer retention and customer engagement.

Pini Yakuel is the CEO and co-founder of Optimove, and can be found at LinkedIn, Google+ and Twitter.

  1. Nice, simple example. Puts the critical importance of testing vs control in terms of finanacial numbers, which changes the entire ballgame.Thanks. BTW: Correct the first $2,100 to $21,000. :-)

    In actuality, the set of all 1,000 of your best customers spent $38,100 this week: the 180 customers who received the campaign spent a total of $36,000 plus the 14 customers who didn’t receive the campaign spent an additional $2,100.

    Thus, the actual gain generated by this marketing campaign is $17,100 ($38,100–$21,000), a far cry from the $40,000 apparent gain we concluded before using a control group.

    Share
    1. Hey Keir,
      Actually, the $2,100 is not a mistake; the control group, which consists of 100 customers, reached a 14% purchase rate, which means that only 14 out of them actually purchased. The average purchase amount of a control group purchaser was $150. Therefore, the total amount of revenue generated by the control group is $2,100 (14 customers x $150). That amount is added to the revenue generated by the test group, and the total of both figures reflects the total amount of revenue the campaign generated. Once subtracting the potential amount that could have been generated without launching the campaign in first place ($21,000), you get the accurate financial uplift.
      Control groups also generate revenue, which we shouldn’t disregard :)

      If you have any further questions around this methodology, I’m happy to answer them.

      -Pini

      Share
  2. Maxim Drobyshev Thursday, November 21, 2013

    Hi Pini,

    Just my 2 cents:
    1. The article lacks statistical confidence intervals calculation and comparison (T-test).
    2. As long as control group was randomized and this distinction is kept secret, you can care less on extra campaign/influence on customers.

    — Maxim

    Share

Comments have been disabled for this post