At the top it looks like this:
1. Measure. Focus on important indicators of what is happening inside your application.
2. Generate hypotheses. Analyze your measurements, think about what can be improved.
3. Test. A hypothesis does not lead to growth until it is tested.
4. Keep knowledge. Learn what makes up your project, how it grows, what influences it.
5. Keep on. Generate new assumptions, act on the knowledge that you received last time.
Now let's look inside. How do we do it in practice and what tools are used:
What to measure is one of the main questions. It is very easy to find yourself in a situation where you collect redundant data. If you count all button presses, all screen transitions within the application and any other actions, then you will not know how to use them all. Most of the collected data will not affect the growth process in any way. And collecting and storing redundant data will be expensive.
To determine what needs to be measured, use the Object➝action model. The essence of the approach is to define all application objects and all possible user actions that can be performed with them. As a result, we get a user interaction map. You can present it in the form of a mind map or a table.
The map will allow you to soberly assess the importance of certain objects in terms of actions. Which of them primarily attract and activate users and affect revenue. By tracking exactly these actions, you will definitely know how you can use knowledge about this to improve the application.
2. Generating hypotheses
Collect your important metrics in the form of visual dashboards. For example, we are using Mixpanel. Each metric can be dedicated to a separate report on the dashboard. This will allow you to focus on specific KPIs that you are currently working on. Dashboards are available to all responsible members of the team. The team holds regular meetings with a focus on ways to improve performance. The team generates hypotheses. Hypotheses must be recorded in a dedicated document. It is necessary to describe in detail what we assume, what solution is proposed, how the hypothesis will be tested and what we want to influence in the end. And also record the current level of metrics now and what indicators we are striving for.
Typically, tests require enough data to be sure that the hypothesis is correct. And yet, when you have little data, for example, a new application, then you can put hypotheses that should significantly affect your KPI. This approach will reduce the time for the test and should still be used with caution. Usually we run the test for 1-2 weeks, this is enough to get data with our traffic.
4. Keeping knowledge
Keep records of how changes to your object affect KPIs and which specific KPIs. Often there is a situation when a test may affect KPIs that you did not plan. Therefore, it is worth taking a broader look at experiments. And here it is important not to run many experiments in parallel. Otherwise, there is a chance of losing the connection between the KPI change and the changes you made to the objects. It will be impossible to understand what changes have affected.
Generate new hypotheses based on the knowledge that you received last time. Continue the circle of improvements based on what you have learned. This will help generate more successful hypotheses and increase the likelihood of success over and over again.
We also share our template for working with hypotheses, where we record results and knowledge. Feel free to download and use it.