(2017-08-09) Torres How To Improve Your Experiment Design And Build Trust In Your Product Experiments

Teresa Torres: How to Improve Your Experiment Design (And Build Trust in Your Product Experiments). If you’ve been following along with the growth of the Lean Startup and other experimental methods, you’ve probably come across this hypothesis format:

• We believe [this capability]
• *Will result in this outcome
• We will have confidence to proceed when [we see these measurable signals]

I like that the “We believe…” hypothesis format is simple enough that it encourages teams to commit their beliefs to paper and encourages them to treat their beliefs as suppositions rather than as assumed facts.

My concerns are that the format encourages teams to test the wrong things and it doesn’t require that teams get specific enough to lead to sound experiment design

I don’t like that this format starts with a statement about a capability. This keeps us fixated on our ideas, whereas we are better off identifying our key assumptions

it’s not the idea you should be testing. You should be testing each of the assumptions that need to be true for your idea to work. So we need a hypothesis format that works for each assumption.

To test Assumption A, you could have a Facebook user review the stories in their newsfeed and share out loud their emotional reaction to each story

this isn’t my only concern with the “We believe…” format

Align Around Your Experiment Design Before You Run Your Experiment

if the objections arise after the experiment was run, it’s difficult to separate valid concerns from confirmation bias.

Our goal should be to surface these objections before we run the experiment. That way we can modify the design of our experiment to account for them.

Invest the Time to Get Your Experiment Design Right

Before you run your experiment, define what you will do if your assumption is supported, if it’s refuted, or in the case of a split test, if the results are flat. If the answer is the same in all three instances, skip your experiment and take action now. If you don’t know how you will use the data, you aren’t ready to run your experiment.

I’ll be sharing a one-page experiment design template with my mailing list members.