How do you Implement Effective Product Experimentation? [Mind the Product]

As product managers, our job is to determine what matters most to our customers. At any given time, there could be 100 really great ideas that need implementing, but realistically we can only complete two or three per month. We have to ask ourselves: which three do we build? How deep do we go? And how much time should we spend iterating on a product that launched last month versus moving onto something new? There’s a high opportunity cost because whichever three ideas we choose to initiate come at the expense of the other 97.

In a world where customers expect more from their interactions with digital companies than ever before, product teams are turning to experimentation to answer these questions accurately. Experimenting lets us measure the impact of the products we’re building quantitatively, by testing them before and after release.

We’re now seeing a shift from the idea of A/B testing as a narrow technique focused on conversion rates and landing pages towards experimentation as a broader mindset of how an entire business runs. By evaluating ideas and making decisions based on data, product managers and their engineers can eliminate the guesswork and avoid wasted time and effort.

In this post, I’ll discuss the foundations of effective product experimentation, and how you can implement it into your business strategy today.

Agile is an Essential Prerequisite

When experimenting, nothing is set in stone. To implement product experimentation effectively, your team needs to constantly refine interfaces, add features and test performance so you can adapt to changing customer requirements in real time — luckily, 94% of you are already practising agile today. Experimentation is the key to agility because it makes your team fearless and gives them the confidence they need to quickly pivot when things aren’t working out.

Make Experimentation Part of Your Product DNA

Today, the most innovative companies, like Amazon, Netflix and Facebook, run thousands of experiments annually. They have embedded experimentation into their culture and use data-driven decisions to implement and create new products. As a team leader, the best thing you can do is to encourage your teams to feel comfortable with failure. Empower all your employees to think and act creatively, and make sure they understand that experimentation is a worthy investment. While you’ll want to have the right infrastructure in place to run experiments and have engineering teams focused on improving that infrastructure, it’s important to remember that experiments can be run by anyone. To make this work, every employee needs to feel like they can get involved and that it’s encouraged to spend time working on projects they are passionate about.

Another best practice is to set realistic experimentation goals. While running thousands of experiments may not be realistic for the size of your team, depending on how many developers you have, determine what it would take to run one experiment per developer per quarter. As you’re building out and defining a Product Requirements Document (PRD), you should always be asking yourselves what experiments you’ve already run that justify building this new product, and what experiments you will run to validate the idea after it launches. By setting realistic experimentation goals, your team will be encouraged to take risks, be curious, and experiment as a day-to-day process. As results from the experiments are reported, teams can use the data to make decisions and understand key learnings and adjust course if needed.

The bottom line is, if you’re going to have an experimentation-first culture, you have to look at every failure as a learning opportunity. Create a hypothesis and test it. If it’s not valid, move on quickly to the next one and learn from it.

Establish Processes and Technology to Make Experimentation Easy

As part of the culture of experimentation, you also need to implement some key processes. One is to embrace gradual rollouts, which mitigate risk. Let’s say you spend nine months building a product. Instead of automatically rolling it out to 100% of your user base and hoping for the best, today we’re seeing more and more teams implement a gradual rollout with the opportunity to roll it back to safety if something doesn’t land as planned. With a gradual rollout, you can slowly introduce a new feature to a small percentage of customers as a way to monitor key engineering metrics like performance and bugs.

For example, Netflix recently began experimenting with airing ads between episodes. Its users expressed concerns about being served commercials, so Netflix decided to gradually test whether surfacing recommendations between episodes will be positively received. By taking this strategy, Netflix is mitigating any risk in making sure this new feature won’t degrade the user experience. On the flip side, Vue Cinema recently received a massive amount of public backlash when customers were entered into a queue on its website to complete transactions, leading them to believe that the website had crashed. If Vue had experimented with this new feature by using a gradual rollout, it would have seen how customers really viewed the experience, and it could have identified what wasn’t working and adjusted its approach.

Another approach to consider standardizing is the painted door test. This kind of test is a great, low-investment way to gather data from your customers, and validate the features you decide to build. Instead of building a feature you aren’t sure your customers will care about, build the suggestion of a feature or solution and measure how many people try to use it. When a customer clicks on the dummy, you can auto-populate a message to say something like “Thanks, we’re thinking of building this feature. Could a product manager get in touch to learn more about your needs?” From there, your team can learn more about what they’re specifically looking for, which can then dramatically change the course of the product direction.

For example, we recently had two different features we were considering at Optimizely, so we ran painted doors for both of them over the same time period and to the same number of people. The results showed that one of them got about 65 people to click on it and the other only got three. While it’s not the be all and end all, this gave us an indicator that we might radically get different value out of these two products.

It’s not uncommon for teams to spend months of time and millions of dollars of engineering effort, all to build an app or a feature that nobody uses. Painted doors are an easy way to avoid this risk. They get you direct feedback from your customers, immediate results, and qualitative insights you can use to re-prioritize engineering resources.

The best products and the most successful companies may seem like they were predestined, but anything worth talking about requires iteration. Product teams that recognize the power of experimentation will be well-positioned for the future. By encouraging your teams to set simple goals, share new ideas and test them at a quick pace, you’ll be able to use experimentation as a way to improve all facets of your businesses.

To learn more about how to implement experimentation into your everyday product strategy, stop by and chat with Optimizely at #mtpcon

The post How do you Implement Effective Product Experimentation? appeared first on Mind the Product.

Source: Mind the Product http://www.mindtheproduct.com/2018/10/how-do-you-implement-effective-product-experimentation/