(2021-12-13) Schmidt Metrics-driven Product Development Is Hard

Daniel Schmidt: Metrics-driven product development is hard. (Data-Informed Product Management) The way that the FAANG companies use metrics to build products is vital to their success. They invest an army of people and homegrown tools to pull it off. My last article, Balancing short-term and long-term product bets, describes Google's process. While the very top companies are great at using metrics, after talking to the hundreds of PMs who signed up for DoubleLoop.com, I’ve learned that almost everyone else is struggling.

Teams risk two primary pathologies that I’ve personally suffered from multiple times:

• Nearsightedness. The team is skilled at moving sensitive metrics, but their optimizations don’t translate to long-term business success
• Farsightedness. The team monitors metrics tied to their company’s long-term business success, but they’re unable to see how their work can move the metrics that matter.

“Good vision,” in the product development sense, is the capacity to make short-term bets that compound to create long-term business value.

While each company has its own vocabulary, I’ve found that Amplitude’s North Star Playbook by John Cutler and Jason Scherschligt provides an excellent synthesis

A key element of the visual model is that the metrics are arranged on a spectrum from “leading” to “lagging.”

Lagging indicators are the KPIs that actually matter for your business; things like revenue and customer retention. Because they are lagging and subject to external forces, product teams often feel helpless in impacting them.

Leading indicators can be influenced by work, like the percent of users that perform a certain action during a session. By themselves, they don’t equate business success.

Your North Star metric is the antidote. It’s the bridge between your leading and lagging indicators

To tie work to business results, you must create a graph/model of relationships between work and metrics.

Here’s how we use the new strategy diagramming tool in DoubleLoop to build our own graph of work and metrics:

Each link in the graph is an assumption that could be wrong or go wrong. You can form hypotheses for how work will move input metrics and for how leading indicators influence lagging indicators. But your hypotheses, of course, can prove false. And a correlation that exists at one point in time might break in the future.

Thus, your map of relationships between work and metrics must be dynamic, not static.

Metrics-driven product development is hard and will always be hard. But there's a lot to be done to make it easier.

We’re building DoubleLoop to be your system-of-record of the relationships between your work and metrics.