Counting On Our Toes
Part 2

Published July 1, 2009

by Ken Will
Technical Manager
D & A Solutions, Ltd.

In the last issue, we explored a number of key points in do-it-yourself customer research. Among the issues covered were the importance of ensuring broad, stakeholder support for the research subject, the different types of research, “nth name” sampling, response incentives, and postage-paid return envelopes.

In Counting On Our Toes—Part Two, we examine two more aspects: the KISS principle as it relates to questionnaire design, and statistical analysis. We should mention, however, that the caveat from the first article applies equally here: these few paragraphs are an overview, not an exhaustive study of all relevant issues. Management intent on doing its own customer research is strongly urged to explore the subject more closely, because the pesky old devil does indeed hide in the details.

In questionnaire design, the acronym KISS means just about what you think it does, absent the pejorative: Keep It Short and Simple. KISS applies to the scope of the project, the length of the questionnaire or survey, and the wording of its questions or “items.” There is an inverse correlation between survey length and complexity on the one hand, and level of response on the other: as the questionnaire becomes longer or more complex, the response rate typically will decline. A rule of thumb for mail questionnaire length is no more than four 8½” by 11” panels, including instructions. Regarding clarity—use short sentences or questions, and “package” them, which is to say, keep like-items together under a common heading and sequenced, if possible, in a logical pattern with a natural flow leading from one subject to the next. When it comes to questionnaire design, remember the “3 Ts”: simplicity, brevity, clarity.

There are, of course, research subjects that lend themselves neither to simplicity nor brevity, and in those circumstances, clarity takes on critical importance… and one of the most common violations of clarity is the compound question. Here’s a compound question that actually appeared in the “Restaurant” section of a resort satisfaction survey the author recently saw in use (the name of the restaurant has been changed, obviously):

Q: Are you satisfied with variety and pricing at the Yum-Yum Restaurant? Yes__ No__ Haven’t gone yet__

Note the potential for confusion—if respondents answer “Yes,” they probably are satisfied with each aspect—variety and pricing. However, if they answer “No,” it isn’t possible to determine whether they are dissatisfied with one, the other, or both. A better approach separates the two aspects while providing greater depth of information, and could look something like this:

INSTRUCTION: On the rating scale beside each question, place a check-mark in front of the score that most closely matches your level of satisfaction. Please note that 1=Very Dissatisfied; 2=Dissatisfied; 3=Neutral; 4=Satisfied; 5=Very Satisfied. If you have not yet been to the restaurant, place a check mark here __ and move on to the next section of the survey.

Q1. How satisfied are you with the restaurant’s Variety? __1__2__3__4__5
Q2. How satisfied are you with the restaurant’s Pricing? __1__2__3__4__5
Q3: What changes would you make to improve the restaurant? __________
_______________________________

With the addition of the 1-5 scale, this approach also creates a numerical basis for analysis of a more sophisticated nature—although we should point out not all practitioners agree that 1-5 scales or so-called “Likert-type” scales are suitably quantitative for advanced statistical treatment. Nonetheless, when (1) there are five or more points on the scale; (2) the intended interpretation of each point is specified; and (3) interval distances are perceived to be equal, the author generally accepts the measure as quantitatively sufficient. With four points, the scale may be thought of as “directionally correct.” Because of between-point information loss, quantitative analysis of scales with fewer than four points can be problematic.

Statistics as a science or discipline is concerned with the collection, analysis, interpretation, and presentation of data. Data is the most fundamental of three levels of increasing abstraction; information and knowledge follow in order, conceptually. Where data may be thought of as the display of a fact, information is the useful interpretation of that fact, and knowledge is the overarching guidance inferred from that information. For real-world summarization, estimation, and prediction, there are two major branches of statistics: descriptive and inferential; and within these branches are four levels of statistical measurement: nominal, such as gender, hair color, political party; ordinal, which includes such concepts such as first, second and third, or Poor, Fair, Good, and Excellent; interval, such as measures of temperature and certain Likert-type rating scales; and ratio, such as weight, height, distance, and so on. With each successive level comes greater analytical depth and testing stringency. Although this sounds complex, and often is, the epistemological witch’s brew that is the science of statistics is dedicated to the task of finding answers for three ironically simple research questions: What are things like? Why are they like that? And, what will they be like if we change something?

Properly conducted customer research is a process of learning and discovery, a fundamental purpose of which is the refinement or development of activities and services that improve the customer experience. Few other industries rely so heavily as our own on the “happy camper.”

An example of such a project is drawn from a survey conducted with a client resort situated in several hundred acres of pristine, heavily wooded hills abundantly populated with wildlife—a truly beautiful natural setting. In the course of the research it was learned that a key motivational driver for participation in the outdoor recreation lifestyle were the feelings of tranquility and emotional relief that came from immersion in the natural environment…but astonishingly, one of the greatest deficiencies in member satisfaction was in member access to that experience! Management was understandably surprised: even though the resort was located within some of the most incredible natural scenery in the state, the survey revealed unequivocally that members were less than satisfied with their “natural” experience!

Further analysis revealed the necessary solution. A professional Naturalist was employed to assist management in the planning and development of a well-informed nature-centric program complete with extensive hiking trails, Naturalist-accompanied tours and overnight camping trips, and numerous other member-interactive events. On the promotional side of the equation, the program was trumpeted in an orchestrated mix of newsletter articles, viral campaigns, and resort signage. The result was doubly gratifying: not only did member satisfaction go through the roof, but maintenance fee revenue spiked appreciably as well.

This exemplifies the earlier paragraph about data, information, and knowledge. The data that began the sequence of events in this particular project was a measured disconnect drawn from paired scales measuring importance and satisfaction; interpretation of that data yielded the conclusion (information) that member achievement of a particular experience was deficient; and the knowledge eventuating from that information served to remind that experience is the product of activity and cannot be assumed to exist simply because underlying resources are abundant. As service providers, we must always connect the dots.

Looking ahead to Counting On Our Toes—Part Three, we will overview specific common statistics that that anyone armed with a modern calculator can produce—and we’ll also take a look at an exotic statistical method that has been known to cause nose bleeds and vapors in the faint-of-heart. You’ve been warned.