Product Experimentation Pitfalls and How to Avoid Them by Jon Noronha [Mind the Product]

Jon Noronha joined the product team for Microsoft’s Bing search engine in 2011 when the product, as he puts it, was “in a big hole”. As the years went by, however, that turned around, and in late 2015, Microsoft reported that Bing had become profitable. Jon and his colleagues at Microsoft attribute that growth to a change towards experimentation. As the then Microsoft GM of Analytics and Experimentation, Ronny Kohavi commented: “The growth of experimentation is the major reason Bing is profitable and its share of US desktop searches nearly tripled.”

Seeing the value of experimentation first-hand, Jon moved from Microsoft to experimentation platform Optimizely, where he is now director of product. Jon learned that though powerful, there are many things that can go wrong when applying this mindset to product. We were thrilled to have him join us at ProductTank San Francisco to share what he’s learned about the benefits of experimentation and how to avoid the common pitfalls that may lead you astray.

Experimentation Pitfall #1: Testing With the Wrong Metrics

At first, Jon and team focused their experiments at Bing on driving more searches per user. The more time a user spends on your page the better, right? Not necessarily when you’re a search engine. After some tests around this metric, they figured out this was actually counter to what users really want and what the market leader (i.e. Google) was focusing on. Instead, they decided to try decreasing searches per session but increasing sessions per user. These metrics in tandem got the user to what they were looking for faster and meant they turned to Bing to get them what they needed.

To avoid this pitfall, Jon recommends:

• Put yourself in the users’ shoes. What if they knew what your goal was? Would they be aligned with that experience?
• Think about your goal as part of the big picture. If this metric went up and everything else remained flat, would this be good for your business?
• Constantly reevaluate.

Experimentation Pitfall #2: Getting Tricked by Statistics

Peeking at results too soon or getting fooled by false positives can mislead your direction. If you’re testing 20 variations with a 5% chance of a false positive, you need to consider that at least one result may be incorrect.

To avoid this pitfall, Jon recommends:

• Involve a professional who knows the numbers. Have a data scientist or statistician join the team.
• Use a testing platform that’s built to consider nuances in the data.

Experimentation Pitfall #3: Testing too few Variations

We tend to focus on A/B testing, but at Optimizely, Jon and team have found that testing many variations is more likely to lead you to a significant winner. On average, two variations leads to a significant winner 14% of the time while five variations leads to a winner 27% of the time.

To avoid this pitfall, Jon recommends:

• If you have the resources, test more than just A and B. This helps the team think out of the box around other possible solutions.

As Jon shared, experimentation is a valuable tool in the product managers’ tool kit, but there’s a lot to consider to make sure it’s implemented correctly. Iterate on your metrics, know your numbers and test! Check out Jon’s talk to hear more stories about product experimentation. He also has a great article on the topic here.

The post Product Experimentation Pitfalls and How to Avoid Them by Jon Noronha appeared first on Mind the Product.

Source: Mind the Product http://www.mindtheproduct.com/2018/12/product-experimentation-pitfalls-and-how-to-avoid-them-by-jon-noronha/