How to Pick the Right Tools for your Experiments by Thomas Leitermann [Mind the Product]

Ready to get started with your experiment? Only need to find the right tool now? But which one to choose? It feels like being in a jungle with tools upon tools – how to navigate this situation?

In this talk from MTP Engage in Hamburg, I share learnings and examples from some of the numerous experiments I ran over the past two years at Inovex. I look at how we can learn from our mistakes, see how easy it is to get started with today’s tools and hear why I believe experimentation is more than A/B testing.

Pack the Treasure Map and Compass

Running experiments for experiments’ sake is a waste of time and money. We should always know WHY we want to run an experiment – what is our goal? How do we know we have achieved our goal? To structure the numerous assumptions and ideas and to think about what success looks like and how to measure it, at Inovex came to rely on the Test Card from Strategyzer.

• It serves as the treasure map for our jungle expedition. On it, we gather all the important information to navigate the tool jungle. The metric defined on the Test Card serves us our compass, informing our decisions.

Pick the Right Machete

When choosing the type of experiment, we have discovered two constraints. Is our hypothesis generative or evaluative? And, how strong does our evidence need to be, compared to the time we have?

When our hypothesis is generative it is either about the customer segment or the solution idea. If our hypothesis is evaluative it is either about revenue and willingness to pay, or acceptance of the solution. This positioning determines the experiment types at our disposal. We can then choose from a wide variety of experiments, for example. from the more than 30 experiment types described in the Real Startup Book.

Once we find an experiment type that matches both our hypothesis and our desired strength of evidence, we need to find time and budget to be able to start.

• For a jungle expedition, this means we now can pack our machete, the experiment type that helps us to cut through the tool jungle. Time and budget can be thought of as food and water for our jungle expedition.

Ready to Go? – Almost

Now we have all the necessities and are ready to go, right? Before going head over heels into the unknowns of the tool jungle I’d offer some knowledge acquired from previous expeditions.

In my experience experiments help to enforce the – hopefully old – mantra “you’re not the user” very well. Rather than build final product features based on assumptions, experiments help us expose those. With experiments we can learn from the people living in the jungle – our users.

In order to expose and learn from our users we first need to connect with them. Automating the recruitment of users for research and testing helps tremendously. There are a lot of useful tools available to find, acquire, schedule, and set up users for research. With those, we can put our user research recruitment on auto-pilot.

• In the jungle, we can navigate using maps and compass – but a GPS device gets us to the goal much quicker.

The Simpler, the Better – and Faster

When looking for tools it’s easy to get down the rabbit hole to find the best, easiest, or latest tool. At Inovex we once spent a number of days rebuilding an experiment on a new tool after the first tool failed us in a small but vital part. So it’s best always to choose the simplest tool. Instead of a high-fidelity prototyping tool – why not use Keynote or PowerPoint? Simple tools also have the added benefit that everyone should know them well. So last-minute changes can be done quickly and without someone becoming a bottleneck.

• If you’re in the jungle looking to chop firewood then you could use a power saw. But an axe also gets the job done and never runs out of fuel.

User-Test your User Testing Tools

There are fantastic tools available to help us understand user behaviour in our experiments, tools like Hotjar, Mixpanel, or Amplitude. They also help with measuring our metric, the compass of our experiments. To be able to rely on metrics we need to make sure to test the tools end-to-end. Only then are we sure they are capable of delivering the metric we need.
At Inovex we learned this the hard way. We ran an experiment for two days before discovering the metric we needed could not be produced by the tool. Only after switching tools and re-running the experiment were we able to answer our experiment hypothesis.

• Before going on a multi-day jungle expedition, it’s probably better to survive one night. Make sure your user testing and measuring tools survive one user.

Iterate Within the Experiment

Once the experiment is running there is nothing to do but wait, right? I’d argue differently. The probability of getting everything right the first time is negligible. I believe the well-known agile motto of inspect and adapt should also be used within an experiment. In one of our experiments at Inovex, we set up ads to drive traffic to the experiment throughout the night, then inspected the results and identified issues in the morning, applied changes in the afternoon and started the cycle again in the evening.

• Making fire with a flint rarely works the first time – but if we inspect and adapt we can get the fire for our overnight stay in the jungle going quickly.

Find a Path Through the Tool Jungle

Equipped with learnings from previous expeditions as well as a backpack with treasure map, compass, machete and supplies I am confident there is a path through the tool jungle.

The slidedeck from this talk can be found here (English) and here (German).

The post How to Pick the Right Tools for your Experiments by Thomas Leitermann appeared first on Mind the Product.

Source: Mind the Product https://www.mindtheproduct.com/2019/08/how-to-pick-the-right-tools-for-your-experiments-by-thomas-leitermann/