(2023-07-16) Cohen Selecting The Right Product Metrics

Jason Cohen: Selecting the right product metrics. I used to believe that “one true metric” (north-star metric) with a smattering of operational indicators was the best way to focus a team on “what matters most.” But I’ve come to believe in a more comprehensive system.

Metrics as a (simplified) value-chain

A product sits in the middle of a chain of events, executed by the team, customers, and peers across all departments. (cf (2022-06-22) Schmidt Why Business Intelligence Falls Flat)

Use color to specify which metrics you are maximizing vs satisficing

plot these events in time, by actor, and by the type of so-called “value” we might measure

The horizontal axis emphasizes that some events happen prior to activity controlled by the product team, and some happen after

Work can—and should—be measured sprint-by-sprint, whereas revenue is a multi-input, lagging indicator of success.

a change in the product can take a while to show up in revenue; individual features often cannot be directly linked to revenue at all

we have placed the metric in context, and understood that it can lag by months or even years, and therefore isn’t a good measure of what’s happening right now.

The most valuable, strategic outcomes are often even more distant from the product team, whether because they are down-stream, or because they are second-order effects for the customer. We control “satisfaction” more than self-motivated external “advocacy,” yet the latter is clearly not only the ultimate measure of the success of the product, but also drives efficient growth. Product teams should take ownership of creating those outcomes, while not allowing those lagging, multi-factor metrics to be the only way we measure progress.

The most valuable thing is for the customer to achieve their own ultimate goal, as defined by the Needs Stack. (2023-06-04-CohenUsingTheNeedsStackForCompetitiveStrategy)

Checklist for great metrics

Defined in normal language: “Customers are using feature X” makes sense.

Defined precisely: In the previous example, you need that technical definition also.

Matches the intent: Often the technical metric doesn’t actually measure what we stated in plain language

Causes action: If the metric does something differently from what we expect, would we act?

Obvious what “good” look like

Measures outcome, not work

Measured easily

Measured frequently Daily is best. One of the advantages of “rolling N days” is that you can update it daily, yet still think in units like “week” or “month” if that’s sensible

Stable over time

Uses common definitions when possible: It’s tempting to invent your own metric

Signal at least 2x stronger than noise