How to Do Testing With a Small Target Universe

Direct marketers in B-to-B environments often find it difficult to conduct traditional DM testing – for good reason. For one thing, most B-to-B campaigns are complicated, with multiple touches and long sales cycles. Simple split testing won’t answer your most important questions, like which touch is essential and which is fluff.

Furthermore, B-to-B campaigns target multiple contacts in single company accounts, so testing can raise thorny policy issues and cause dissatisfaction not only among customers but also with the sales force. It’s not easy when a sales rep comes screaming into your office that your offer test has created confusion in her territory.

But the major reason why DM testing is tough in B-to-B is the size of the typical target universe. When your entire prospective audience is smaller than 10,000 accounts – a not uncommon situation – test-and-rollout strategies are likely to be pointless.

So, how do you take advantage of the power of testing when universes are too small to support traditional split methods? Let’s look at four types of work-arounds.

1. Eliminate Outliers
When you cannot test to a sample and roll out to the whole, you are in effect campaigning to the whole every time. So what can you do to improve the odds of going out to the target with your best shot? Use pre-testing to eliminate the least-powerful offers and creative approaches.

Some of the pre-testing techniques that work for B-to-B marketers include:

  • Using email and phone to pilot offer and creative ideas, and then apply those results to other media, like the mail.
  • Focus groups, whose opinions can eliminate the least promising ideas while also providing deeper insight into the motivating factors behind respondent behavior.
  • Online surveys via low-cost tools like Zoomerang and SurveyMonkey.

2. Take Manageable Risks
When you can’t do pure testing, with statistically projectible samples to provide reliable predictions of campaign results, one alternative is to adjust your expectations, accept an increased amount of risk, and then seek ways to mitigate that risk.

Here are some approaches that can work:

  • Lower the confidence level required for your tests. Mark Klein, CEO of Loyalty Builders LLC, points out that reducing your expected confidence level from the DMer’s traditional 90-95% to something in the 60% range can make dramatically smaller test quantities productive, or at least directional. “This kind of testing, while more prone to error, still improves your chances of hitting a winner with a small target universe, which is the whole point,” says Klein.
  • Campaign sequentially. When you are running campaigns to the same audience over time, the results will mirror what you’d get from a split test. There is some risk, since you are introducing time as a variable. But, says Andrew Drefahl, director of customer insight and technology at Hunter Business Group, “Rapid iteration becomes a learning process. You learn by failure, developing rules of thumb that you can then apply to other campaign situations.”
  • Make reasonable assumptions. For example, take what works in one medium, like the phone, and make the assumption that it will also work in the mail.

3. Get the most out of what you have
Even if you don’t have a universe large enough, some tools are available for taking advantage of what little you have.

  • The “Power Test,” pioneered by The Hacker Group, and now in wide use by such skilled DMers as The Kern Organization provides a methodology for generating a control direct-mail package from scratch using only 50,000 names. Essentially, the Power Test suspends the need for each test cell to be statistically projectible on its own, but lets you read results in aggregate across several lists. This way, with only 2 test runs of 25,000 each, a control package can be culled from 2 offers, 3 creatives and 5 or more test lists. A very neat technique indeed.
  • The enterprise-wide control group. As advocated by Richard N. Tooker, author of The Business of Database Marketing, a company can set up a single control group for multiple campaigns over time by establishing an ongoing, standard control group made up of, say, 5% of the customer base. The purpose of the corporate control group, which receives no marketing communications at all, is to assess the value of the direct marketing program as a whole. But it may also be used as a benchmark for particular campaigns, if the campaign goes to the entire balance of the universe. Tooker cautions that if the campaign is going out to a smaller segment, the portion of the corporate control group that is comparable may quickly become too small to be valid.
  • Test at the contact level, but read results at the site level. Andrew Drefahl suggests a neat way to enhance the size of your campaign universe by contacting as deeply as possible into an account, but then measuring campaign success in aggregate, by how well the account as a whole accepted your value proposition. Drefahl recommends using the phone to measure this acceptance, or “uptake,” since it allows you to probe on the whos and whys of the response. “Using the phone is critical to the test and learn approach, which is often the only way to open doors in a small B-to-B audience,” he says.
  • Steal from your competitors. Jim Obermayer, founder of the Sales Lead Management Association, suggests that B-to-B marketers take a page from DM creative directors, who have learned over decades the value of the “swipe file,” which stores competitive control ads based on how often they see the piece in the market. “If your competition has figured out what’s working, why not reap some benefit from the other guy’s investment,” says Obermayer.
  • Offer the hottest premium. Business people respond well to free stuff, and new tchotchkes are coming on the market all the time. If you build a strong relationship with an ad specialties vendor, you can be first in line to use the hot new item. “Remember those fans a few years back, where the brand name was spelled out in the air via electrodes?” says Obermayer. “Everyone had to have one. There’s no staying power in this strategy, but you will certainly be giving the market your best shot.”
  • Use your house file as your testing medium. As Stephen R. Lett, president of Lett Direct, points out, “The higher response rate from your house file enables reliable testing with a smaller universe. You can then take these results and apply them to your prospecting campaigns. It’s not perfect, but it’s better than no testing at all.”
  • Run an A/B split test in print. Trade pubs often have a circulation large enough to permit valid split testing – offer, headline, or copy platform, for example. The results can then be applied to other media.
  • Swing for the fences. With small universe sizes, you don’t have the luxury of testing nuances. So test the changes that will have the most impact, advises Lee Marc Stein, the veteran copywriter.

4. Leverage the Internet
Internet media offer B-to-B marketers several brilliant new options for improving their campaign results.

  • Pre-testing using email and banner ads. A/B splits in Internet media do little to enhance the size of your universe, but what they do provide is speed and convenience, making testing cheap and easy. So take abundant advantage of split testing in headlines, offers, subject lines, and other variables that can then be applied to your campaigns in other media.
  • Website-based multi-variate optimization. Thanks to free tools like the Website Optimizer in Google Analytics, marketers can set up multi-variate tests on critical web pages like registration forms, order forms, and landing pages – anywhere that a response is requested.

Back to Articles and Columns »