(2018-07-02) Conservation Of Intent The Hidden Reason Why Ab Tests Arent As Effective As They Look

Andrew Chen: Conservation of Intent: The hidden reason why A/B tests aren't as effective as they look

this is an infuriating startup experience: You ship an experiment that’s +10% in your conversion funnel. Then your revenue/installs/whatever goes up by +10% right? Wrong :( Turns out usually it goes up a little bit, or maybe not at all.

Why is that? Let’s call this the “Conservation of Intent.”

For all your users coming in, only some of them are high-intent. It’s hard to increase that intent just by making a couple steps easier – that’ll just grow your low-intent users

First, understand what’s really blocking your high-intent users.

You obviously get real net incremental increases on conversion the further down the funnel you go. By that point, the low-intent folks have burned off. You’re closer to the bottom line.

When you focus on low-intent folks, you’ll have to get creative to build their intent quickly. Things like being able to try out the product, having their friends into the product – these are the “activation” steps that generate intent. Here’s a great place to start – a highly relevant essay on getting users more psych’d, guest written by Darius Contractor from the Dropbox growth team.