Stop Overthinking Your Survey Metrics

Debating the survey method is a tactic to explain losing

Nearly every competition has some different scoring system that may be straightforward or confusing. In sports like soccer or hockey, each score is counted as an increment of one. In football, a team gets six points, but can get an extra point or two. Unless they kick and get three points. Then there is tennis, where each score is in increments of 15, except after 30, then it is 10.

Regardless of the actual scoring system, it doesn’t take too much for any observer to figure out who is winning. It’s the one with more than the other team (okay, golf is a funky outlier). Similarly with customer experience surveys, regardless of the metric or method, it is easy to see who is doing well.

Most CX professionals have found themselves in a debate over the ultimate customer experience metric. Some fall into the “Net Promotor Score” camp (evaluating loyalty by estimating “promoters” and “detractors” based on the respondent’s likelihood of recommending). Others prefer a “Customer Satisfaction Score” (identifying the percent of respondents identifying with the top end of a satisfaction scale). And other groups prefer the simplicity of a five-star average. All these have their merits and drawbacks.

There is no right answer to choosing the right survey method for all companies. There is a wrong answer, though. It is the one that comes from spending too much time debating which survey to choose or changing methods in the middle of a customer experience initiative.

In the end, it really doesn’t matter what metric a company uses. Whatever metric is chosen will reveal if the company is improving and which teams are doing well (and which ones are not).  What does matter is sticking to something and improving over time. The best thing that any company can do is rally behind a method that makes sense and is easy for everyone in the organization to understand (including what is going right or wrong). As long as there is a benchmark and improvement can be shown while aiming for excellence, the company can be successful.

Alignment and engagement are highly dependent on the metric. For that reason, companies are advised to evaluate the following considerations in choosing a customer experience metric:

Simplicity

The more math or calculations that an employee or team needs to do in order to determine their score, the more work it is for employees and less likely that they will follow the measurement or engage with the work. For teams to engage with the data and respond through action, the metric needs to be something that comes easily or automatically to them. If more complicated calculations are necessary (e.g., calculating promoters and detractors) then the back-end work needs to be done to automate the results for front-line teams or managers to pull the data themselves. Teams are less likely to be engaged if reporting is centralized and reported to them periodically.

Timely

The more a person can see the data, the more they are likely to engage in doing something about it. Think about a person on a diet. They will likely jump on the scale daily to see if the healthy dinner or long walk the day before helped. Similarly, employees need to be able to correlate the work they do with improved numbers. If they need to wait many weeks for the metrics, they will not be able to connect their efforts to the results. Even if time must pass to get a representative sample for reliable results, feedback from even a few responses will help engagement. It is better to err on the side of employees having immediate feedback, but only holding them accountable to results after a reliable sample is collected.

Relatable

Both leadership and employees need to be able to clearly understand how their actions can impact the scores. They need to be able to see that the metric is a reflection of their specific work, they must understand it, and they need to be in control of improving the score. For example, it does not make sense for a hotel’s valet staff to be accountable to the room’s cleanliness. The more that a metric is a combination of multiple business units, the less likely a specific unit will be engaged in it.

It can be tempting to hold an entire organization to a combined metric in an act of teamwork and shared responsibility. However, teams will quickly assume that they are unable to control the outcome, blame other teams or be unaware of their influence. They will quickly disengage with the initiative. Accountability will come through transparency and applicability. When comparison of similar teams can be published, it will further increase accountability.

Qualitative

Companies will often find that customer comments will be more actionable than a quantitative metric. Which of the following is more effective for driving change: 1) Receiving a score of 7.2 out of 10; or 2) a number of comments stating, “I had to wait more than 30 minutes”? The former leaves the company guessing on what to do whereas the later will tell you that there is a wait time issue and people’s tipping point for frustration is about 30 minutes.

An overall metric is still essential to understand the overall experience, comparing to other business units, monitoring trends over time, and how elements of the experience impact the overall experience. But qualitative feedback should be incorporated into surveys when possible.

Consistent

It is essential that a company pick a metric that makes sense for their organization and how they are able to best manage it. Once that metric or method is selected, it needs to be consistently used and survive any leadership changes that take place. The worst thing for building momentum and action is to allow teams to advocate for changing the scoring methods and starting over.

It is common for companies to change methodology for a few reasons. One example is that companies can fall into a trap of overthinking the data. It’s much easier to keep looking at numbers rather than doing the hard work of operational and cultural change. Changing the methodology is a hope that the actual customer experience is better than it has shown because of using the wrong formulas. Another reason for switching directions is that it is common for teams to get wise to which data will make them look best and advocate for whatever they can use to manipulate the data. These debates usually happen as tactics to stall the work to improve or excuses for failure. It is much easier to blame the referees for losing.

Comparable

One of the arguments companies have for a particular survey methodology is using a similar methodology as competitors. There can be value in knowing a company’s relative performance. For example, there are industries like healthcare where getting an average satisfaction of 7 out of 10 (a relatively average score) might actually put a health system in the top 90th percentile if all are measured by the same metric. But that is just another way of a company trying to look better than it is performing. Customer expectations are formed by the customer’s experiences across industries. It is better to be measured objectively by a customer based on his or her expectations rather than being graded on a curve to claim the company is slightly better than poorly performing competitors.

When it comes to comparability, it is more important to consider how internal departments can be compared and evaluated. If the company has multiple divisions, locations or departments that are similar, the same metric must be used to best compare, learn lessons and ensure accountability.

The ideal survey

Given that the 4Ps of Customer Experience™ guide any customer’s satisfaction, it is recommended that a company create a brief survey that covers some overall summary metric, then one question in each of the 4Ps (People, Process, Product and Place). Ideally, there is an opportunity to provide both a quantitative rating on each and the option of providing context through comments. This will allow the company to have insight into which of the 4Ps of Customer Experience™ are impacting the overall score and where improvements can be made.

Simple and effective feedback surveys that are easy and quick for the customer are the key to improved customer experiences. Choosing metrics that build alignment and engagement will lead to improved accountability, visibility and action.

Michael Pearce is a managing principal at Hundred Ten, LLC