Interviewer: Walks in the room. After the initial introductions, “ I would like to discuss about product metrics”
You: Sure! (Internally you are just hoping that the problem is a easy one :))
[Interviewer will discuss about a specific feature or real product example and ask you to choose the most important metrics and how to measure them?]
You: Ok. Here is what I would do
Let’s look at different scenarios and breakdown the steps involved in answering product metric questions in the program manager interview.
Most of the Product manager interviews usually include a section on metrics. One of the most important ways to tackle metric questions is to pick a framework and tackle the most important product metric.
What are product metrics and why are these questions included in a product manager interview?
Product metrics are data points that a PM tracks and analyzes to assess the success of their product. Metrics and Key Performance Indicators (KPI) help all stakeholders in an organization determine how customers are interacting with a product, the value it brings a company and how it can be improved upon.
Some examples of product metrics include monthly recurring revenue, paid conversion rate, churn rate are all examples of quantifiable product metrics. There are many product metrics to choose from.
Questions to ask include, user adoption rate, customer retention rate, or churn rate. Some metrics tie back to business goals such as cost to acquire new customers, lifetime value of the customer, or the average revenue from the customer. PM needs to check if the metrics align with the future direction of the product in the next 2-3 yrs. Metrics for future direction include sessions per user, number of active users, and sustained feature adoption.
The metric interview questions will be used to judge your ability as a PM to understand the product or feature and define metrics. Here’s an example question: “What metrics would you use to determine success for Facebook Sponsored Posts?”
As a PM you should be able to understand the business value and the feature of sponsored posts and then define metrics such as total impressions, clicks, Likes, the net return on ad spend, or the number of comments. You should be able to rank these metrics and select the most important ones.
Framework to answer interview questions:
We will approach the interview questions with the GAME framework. GAME stands for,
Start with defining the product or the feature. For example if you are asked a question to define metric for Twitter timeline, you should start with defining the Timeline feature
Twitter is a social network where a user can post short posts or “tweets” up to 280 character limit. Earlier it used to be 140 characters and now Twitter has changed the limit to 280. Users can also post, like, retweet and share tweets from other users in the network. Timeline consists of top tweets from other twitter accounts followed. Timeline updates and refreshes every time the user logs in.
After defining the product, start defining the GOAL of the metric. In case of the Twitter interview question, Twitter timeline is one of the most important feature of Twitter. Timeline is a primary driver for user engagement, and a strong revenue generator. Twitter will test new features on the Timeline also. So, how do you determine what the most important objective is?
There’s no single right answer to that question but you could begin by saying, “My understanding is that there are several use cases for the Timeline including engaging users, generating ad revenue, and testing new features. I am going to assume that the user engagement is the most important goal and proceed to define the metric. Is that what you have in mind too?”
Before jumping into solution space, I will confirm that the interviewer agrees with the specific Goal that I selected for the success of the product. Next step is to define the Action
What are the actions that a user will take when using the product or feature? I will list all the actions taken. Writing this list provies two things, One that I as a product manager understand the product and the use cases. The second point is that listing the actions makes it easy to fous on the metrics necessary.
We will visit the Twitter Timeline example again. What are the actions that a user will take. The user will create a tweet, view tweets, comment, like or share tweets. The user can also follow other users and topics.
You can confirm with the interviewer that you are sticking to the basic user actions and not looking at Twitter followers or topics. The rule is to keep it simple and not over complicate the actions. Remember that you only have 20 to 40 minutes to walkthrough the entire metric interview question.
Next, you need to prioritize your list of actions. Remember to use the goals you established in the previous step to explain your thinking. For instance, you might say something like, “There are a number of user actions which reflect engagement but I think the three most important ones are likely to be posting, commenting, and sharing.”
Once you have a prioritized list of actions, it’s time to define associated metrics for each one. In this step it’s important that you define the metrics you will measure with precision.
We had picked user engagement as the Goal and the actions taken by the user earlier. Engagements represent the number of times that a given Tweet was engaged upon by a viewer in an organic context.
Engagements include, but are not limited to, Retweets, Favorites, Replies, URL Clicks, Hashtag Clicks, Mention Clicks, and Media Views.
We will define FOUR metrics for this goal
- Favorites: The number of tweet that were favorited by the user
- Replies: We count the number of replies, by an user and the total replies that a tweet gets
- Retweets: The number of tweets that were re-tweeted by the user
- Views: The total number of views for any tweet (tweet was clicked)
The last step in the metric process, is Evaluations. I will conclude the interview by evaluating the metrics chosen. I will also highlight the trade-offs and limitations.
In case of the the Twitter Timeline example, I would list the evaluations as follows:
I would look at the number of favorites, replies, retweets, views per thousand sessions. The metrics measured would give a good idea on the engagement for an user.
The last key step is to show that you understand the strengths and weaknesses of your recommended metrics, by listing the tradeoffs or limitations. We will list the evalautions for the Twitter Timeline example below.
I consider replies as high user engagement whereas favorites, retweets and views are lower user engagement. Not all users will reply to tweets but will still use the product frequently. We did not consider the total time spent by the user per session. Time spent is hard to measure if the user does not log off every time.
Always use a framework to answer metric questions. Do not jump into solution space immediately.
In this blog, we used the GAME framework to answer interview questions. I will show another example using the HEART framework (Google) to answer metric questions.
I have outlined a step-by-step method you can use to solve metric questions. I encourage you to try solving a couple of the sample questions using the steps defined in this blog.