(2014-09-05) Torres The14 Most Common Hypothesis Testing Mistakes Product Teams Make And How To Avoid Them

Teresa Torres: The 14 Most Common Hypothesis Testing Mistakes Product Teams Make (And How to Avoid Them). Teams who invest in research are quickly finding that their experiments are only as good as their hypotheses and experiment design.

1. Not knowing what you want to learn.

In the product world, we can experiment at different levels of analysis.

Too often people test all of these layers at once

2. Using quantitative methods to answer qualitative questions (and vice versa).

Quantitative research is great for uncovering how a large population behaves, but it can be challenging to uncover the why behind their actions.

3. Starting with untestable hypotheses

This might be the most common mistake of all.

Design A will improve the overall user experience.

4. Not having a reason for why your change will have the desired impact.

The problem with these types of experiments is that they increase the risk of false positives

5. Testing too many variations

More variations leads to more false positives

6. Running your experiment with the wrong participants

Often times the who is implied in the hypothesis. Do the work to make it explicit so you don’t make these errors.

7. Forgetting to draw a line in the sand.

*For quantitative experiments, draw a hard line in the sand. How much improvement do you expect to see?

Draw the line as if it’s a must-have threshold. In other words, if you don’t meet the threshold, the hypothesis doesn’t pass.*

8. Stopping your test at the wrong time.

Determine ahead of time for how long to run your test.

Use a duration calculator.

9. Underestimating the risk or harm of the experiment

10. Collecting the wrong data

What would the data need to look like for me to refute this hypothesis?

11. Drawing the wrong conclusions

Before you draw a conclusion, you need to ask, “What else could explain this result?”

12. Blindly following the data

13. Spreading Yourself Too Thin

Learn a method inside and out before moving on to the next one.

14. Not Understanding How the Tools Work