Good enough data

It’s better than good. It’s good enough.

- Community (sometimes tv shows make sense)

You’ve likely encountered awareness campaigns about the dangers of air and water pollution. Nutrition programmes to combat obesity and diabetes. Vaccination drives to ensure that children are properly inoculated against debilitating illnesses. How do governments decide where to deploy limited budgets so they have the greatest impact on public health? This is not something we think about much (or at all), but accurate and consistent data on disease and death rates are critical for policy makers to prioritise investments effectively.

One approach, is to start by collecting all the data. This was the ambitious methodology adopted by medical doctor and economist Dr. Chris Murray, for the Global Burden of Disease studies. His intent was to systematically and comprehensively tabulate the world’s illness and mortality rates¹.

Product Managers often find themselves in a similar position – determining the best way to allocate limited resources for maximum impact. And it can be tempting to emulate a similar approach in the quest for the “perfect” data-driven decision – investing significant time and effort in instrumenting and collecting all available data, trying to improve the precision of these measurements, and finally building complex dashboards to analyse and visualise every detail. Trying to follow the user journey of each user to “solve” all of their problems, though, is unnecessary, and often counterproductive.

Focus on purpose, not perfection

Instead, first define your objective, and then come up with an acceptable metric to measure it. Once that’s clear, establish a metric that’s “good enough” to measure progress toward that goal.

Acknowledge that all metrics are approximations, and that they will fall short (as numbers always do), when they try to represent reality. Also ask yourself if a more precise measurement will really make a significant difference to outcomes. If not, move on. You can always revisit the metrics as your product and consumers evolve, but frequent churn with definitions, and parsing increasing amounts of data in ever more complex ways will invariably cost you more than it’s worth.

Avoid the trap of over precision

I’ve seen Product Managers obsess over whether churn was 35% or 36%, when the distinction has no material impact on what they’re going to do about it. A far better use of time is to apply consistent methodologies and definitions, observe trends, and focus on actionable insights.

Similarly, customer support ticket volume tracks the number of support requests or issues reported. General trends and patterns can be valuable, even if individual ticket data isn’t completely accurate or comprehensive.

Conversion rates by ad, or media, are another great example of a product metric that is often tracked uncompromisingly, without much to show for it. While the metric itself is important, acknowledging that it is difficult to correlate the extent to which a specific ad contributed to the customer’s decision is equally important. A purchase might be influenced by hearing about the product from a friend, or interrupted by mundane events, like a doorbell ringing mid-payment. Attempting to ascribe a motive to all of the users’ decisions is not just impractical, it’s wasteful. Decide how good your estimates need to be, and live with them. If you find yourself with extra time, here’s a better use: talk to your users instead.

In each of these cases, striving for accuracy is valuable. However, actionable insights can often be derived from approximate data, allowing teams to make informed decisions without needing perfect information. Not only does this ensure that you’re optimising the dollars spent on instrumentation, storage and analysis, it also trains you to focus on what really matters – impact.