ARTICLES BY VSI
Using an A/B Split Test to Maximize Coupon Promotions
Lowering your coupon face values might cut the cost of your promotion significantly, but it might also cut its effectiveness. How can you be sure before you make a drastic move? When faced with the challenge of testing new ideas like that, savvy marketers often turn to a valuable market research tool: A/B splits. This technique can be tailored to many different kinds of variables and the resulting data can be used to guide strategy and tactics on a much larger scale.
Defining A/B Splits
Most everyone is familiar with the foundation of a basic experiment, or market test. Introduce an element in one market, and do something differently in a “comparable” market. The difficulty, of course, is that no two markets are ever exactly the same—it is nearly impossible to determine if variation between markets is due to the variable being tested or very slight market differences.
An A/B split test removes that uncertainty from the experimental design. The A/B coupon test offers one coupon to half of the market, and a different offer to the other half. This limits concerns that the test is invalid. Because of similarities in market (both offers distributed in the same market) and media (both offers distributed through the same newspaper), an A/B split allows the differences in redemption to be clearly acknowledged.
Before You Start
It is vital for companies to develop a clear understanding of their needs and goals prior to designing an A/B split. Proper preparation is necessary to ensure a valid experiment, and actionable results. Planning should include determining what variable(s) should in fact be tested, taking into account market-specific factors and trends that may influence consumer response. To get actionable data for future use, it is important to explicitly outline the strategic goals of the A/B split (i.e. what will be learned), and to develop a plan to carry out the split and analyze its results.
Creating A/B Splits
There are several ways to set-up an A/B split, and many ways to use the results. Splits can be achieved by collaborating directly with placement companies, but many leading companies work with coupon consultants like VSI Targeting for testing. Consultants can provide key insights by evaluating markets to help determine the ideal location for the test; their expert knowledge of market-specific conditions, such as coupon doubling policies, ensures the validity of the test. Coupon consultants can also be invaluable in analyzing the results, and implementing the findings into more effective promotions. Most consultants are familiar with the many options available for an A/B split, given a company’s needs.
There are three different kinds of splits, which vary in cost and data reliability: 50-50 splits (also known as “imperfect”), palette splits (also known as “splits by lift”), and “perfect” splits. The three options may not be available in all areas, however; companies should check with their placement vendor to see what types of A/B splits are offered. 50-50 splits are the simplest and cheapest option. With a 50-50 split, the newspaper inserts Offer A into the first batch of newspapers printed and Offer B into the latter batch. Palette splits are middle-range, both in cost and accuracy, as the offers are alternated for each palette of newspapers instead. An issue for both of these methods is the homogeneity of the market being tested and the dispersion of the diversity of the population. For example, a 50-50 test may result in Offer A being distributed to the western part of a city and Offer B being distributed to the eastern part.
If there are significant demographic or behavioral differences between these areas, the resulting data will be skewed accordingly. Palette splitting can result in similar statistical problems, albeit to a lesser extent. The “perfect” split addresses those problems, although at a higher cost. One copy of the newspaper will be given Offer A, the subsequent copy will be given Offer B. Clearly the labor-intensive aspect results in higher cost, but it addresses even significant intra-neighborhood variation.
Evaluating A/B Splits
Much like any other experimental design, A/B splits carry unique benefits and limitations that marketers should be aware of. They are summarized in the exhibit below, and discussed in the following sections.
Benefits of A/B Splits |
Concerns about A/B Splits |
|
|
Concerns about A/B Splits
There are characteristics of A/B splits to be cautious of, as there are with any experimental design. While the A/B split does protect against inter-market variation, very few manufacturers do business solely in one market. A single split is of limited utility for extrapolating results to a national scale. For this reason, it is often recommended that marketers perform multiple A/B splits in different markets, and that they recognize the limitations of those results. As with any statistical sample, variation may occur, particularly as markets diverge from the sample in demographics, media penetration, product penetration, and other factors. Experimental results should be used to guide strategy and planning, but can never be depended upon as entirely accurate.
Additionally, companies may encounter difficulties when many different variables are tested simultaneously. It is therefore recommended that marketers test one variable only, and that one of the offers be similar to the “standard.” One of the two offers should have a benchmark of previously established data that can be compared to the redemption for the test results; this will minimize the complexity of the analysis.
Another potential obstacle in analysis is the presence of differing double coupon policies in many markets. As previous VSI research has revealed, double coupons can have a significant impact on coupon redemption rates. Marketers should be aware of policies in the areas being tested, and account for that if in fact lower face value coupons redeem higher than expected.
Benefits of A/B Splits
The benefits of A/B splits, regardless of the specific method chosen, can be significant. It forces marketers to focus on a single measurement tool—cost per unit moved—which simplifies post-mortem analysis. A/B splits are also surprisingly flexible; they can be used to test not only face values, but also creative execution, purchase requirements, products featured (e.g. A 20oz size versus a 12oz size), etc. The adaptability of the A/B split allows for marketers to test for one element and to use one common measurement to gauge its success without market variation affecting the analysis.
A/B splits can also be used regardless of the position of a company within a market. Manufacturers with low brand trial in a given market, for example, may want to experiment with higher face values to see if an impact can be made. Conversely, manufacturers with some market penetration may want to see if they can reduce their face values and promotional costs without adversely affecting redemption rates significantly.
A snapshot of the findings of a recent A/B split, conducted by VSI Targeting for a leading CPG company, exemplifies the many benefits of this experimental technique. Over the course of several tests in multiple markets, VSI was able to provide its client with detailed recommendations regarding the ideal offer for several markets including attributes such as face values, the number of required purchases and creative execution. This enabled the client to discontinue ineffective practices and tailor promotions to particular areas, significantly lowering cost of the promotion per unit moved, without sacrificing reach in many cases.
A/B splits remain a potent tool for maximizing coupon promotion. Whether conducted directly or through a coupon consultant like VSI Targeting, A/B splits should not be overlooked for the reliability, simplicity of measurement, and flexibility they can provide.
