Running live experiments at Justdial (Step-by-step guide): Product Management
Do you remember a time when you launched a product you were certain would be a hit, only to have it fail? Well, you’re not alone. The journey to find the product market fit can be difficult. In this article, I would be discussing how to run experiments to validate your hypothesis and launch successful products.
Defining the Problem:
Identifying the correct problem to solve is crucial and may seem straightforward, but it is of utmost importance. A well-defined problem serves as a guiding light throughout the process. It helps you to understand what problem your product or feature is designed to solve. Just, don’t skip this step.
A Problem Well Stated is Half Solved - Charles Kettering
Justdial’s mobile websites had a substantial number of daily users, but very few of them logged in, posing a challenge for both users and businesses. For businesses, it is difficult to monetize non-logged users, and for users, logged-in results are more accurate, providing a better experience. Therefore, the problem at hand is how to increase new logins.
Making the hypothesis:
With your detective hat on, it’s now time to formulate a hypothesis. Utilize your understanding of the problem and your target customers to create a hypothesis on how your product will solve the problem. This is your opportunity to tap into your creativity and make informed guesses, utilizing data and user insights, on what will be the most effective solution.
The importance lies in being purposeful and well-informed while formulating your hypothesis. Having a clearly defined problem and a robust hypothesis, you are now prepared to progress to the next step, which is designing your experiment.
At Justdial, I formulated several hypotheses. One of these was that an improved login design would result in increased logins, while another was that users could be encouraged to log in through subtle nudges on new pages.
Designing the Experiment:
The time comes to build your experiment and put your hypothesis to test and validate it. There is a variety of experiments that can be carried out, such as creating a minimum viable product (MVP), administering surveys, organizing focus groups, or performing A/B tests. The essence is to select an experiment that will provide the necessary data to support or refute your hypothesis.
While designing experiments, three things are very crucial are:
- Make it simple, the more you make it complex harder it would be for you to analyze. Never run multiple experiments together, you never which worked and which didn’t.
- Focus on the right audience, as It is crucial to concentrate on the appropriate audience and guarantee that you have a sufficient and representative sample size for your experiment in order to validate your hypothesis.
- You must know the numbers, and make sure you set up the right events or data points to measure whether it’s a success or failure.
At Justdial, I utilized the power of minimum viable product creation to assess successful and unsuccessful strategies and later on A/B experimentation. In terms of design, we developed a new pop-up design featuring a prominent call-to-action and implemented it in multiple locations where our previous pop-up had been displayed.
Another hypothesis we had was to encourage users to log in on various new pages. To test this, we began displaying a login pop-up on the home page, with specific criteria for when and to whom the pop-up would be displayed (The details of this logic can be a story on its own :D)
Later on, we also ran an A/B experiment too, we tested two login pop-up designs on the home page. One design displayed both the name and number fields, while the other first asked for the number and then the name. By experimenting with both designs, we were able to observe and analyze the results.
Execution & Analysing results:
You’ve made it to the last step and it’s time to execute and measure it. Make sure, you document the whole process & learning, such that you can always refer to it when needed.
Another aspect of experimentation is to be mindful of past metrics. For example, if you aim to enhance logins, it is important to have an understanding of past login numbers in order to determine performance. Additionally, it is essential to monitor secondary metrics, as they can provide insight into the unintended consequences of your experiment. A seemingly successful result based solely on the primary metric may not accurately reflect the overall impact.
When evaluating your results, maintain an open perspective and be willing to revise your hypothesis if the data indicates it is incorrect. This is a repetitive process, so don’t be disheartened if your first experiment does not yield the desired outcomes. Utilize the data you have gathered to guide your next hypothesis and continue experimenting until you discover the right fit between your product and the market.
Results: Both experiments showed success, with the new login design resulting in a 6% increase in new logins, and the homepage pop adding an 8% new logins. Overall, we observed an average increase of 14% in logins per day. 🎉🙌
FYI, the A/B experimentation I talked about above is still in process & I’m waiting for its data numbers too 🤓
Note: Additionally, I also tracked secondary metrics such as engagement and time spent to ensure that displaying the pop-up on the home page did not have any adverse effects. However, we observed that these metrics remained almost the same, confirming the validity of all our hypotheses.
And there you have it! You’ve successfully completed the journey. I hope these tips were valuable in assisting you in your quest to find the right product-market fit. Keep in mind, the journey can be unpredictable, but with a systematic approach and a willingness to experiment, you will be well on your way to success. Best of luck with your experiments!
P.S. This is just one of the many experiments we implemented to increase new logins, stay connected to know more about these. Say hello here at my Linkedin profile. 👋
