How to use Google HEART to answer metric interview questions

How to use Google HEART to answer metric interview questions

We will show to apply a framework to answer YouTube Product Manager Interview question.

Interview Scenario:

Interviewer: Walks in the room. After the initial introductions, “ I would like to discuss product metrics”

You: Sure! (Internally you are just hoping that the problem is a easy one :))

[Interviewer will discuss a specific feature or real product example and ask you to choose the most important metrics and how to measure them?]

You: OK. Here is what I would do

Let’s look at different scenarios and breakdown the steps involved in answering product metric questions in the program manager interview.

Most of the Product manager interviews usually include a section on metrics. One of the most important ways to tackle metric questions is to pick a framework and tackle the most important product metric.

What are product metrics and why are these questions included in a product manager interview?

Product metrics are data points that PM tracks and analyzes to assess the success of their product. Metrics and Key Performance Indicators (KPI) help all stakeholders in an organization determine how customers are interacting with a product, the value it brings a company, and how it can be improved upon.

Some examples of product metrics include monthly recurring revenue, paid conversion rate, churn rate are all examples of quantifiable product metrics. There are many product metrics to choose from. Questions to ask include, user adoption rate, customer retention rate, or churn rate.

Some metrics tie back to business goals such as cost to acquire new customers, lifetime value of the customer, or the average revenue from the customer. PM needs to check if the metrics align with the future direction of the product in the next 2-3 yrs. Metrics for future direction include sessions per user, number of active users, and sustained feature adoption.

The metric interview questions will be used to judge your ability as a PM to understand the product or feature and define metrics. Here’s an example question: “What metrics would you use to determine success for Facebook Sponsored Posts?”

As a PM you should be able to understand the business value and the feature of sponsored posts and then define metrics such as total impressions, clicks, Likes, the net return on ad spend, or the number of comments. You should be able to rank these metrics and select the most important ones.

The biggest mistake that you can make is to jump into the solution space immediately. The candidates who are successful will always clarify the product or feature with the interviewer and then use a FRAMEWORK to answer this question. I have a blog post that describes the TOP THREE frameworks for measuring metrics.

Framework to answer interview questions:

We will approach the interview questions with the HEART Framework. HEART stands for,

WHAT IS THE HEART FRAMEWORK

HEART (developed at Google) stands for Happiness, Engagement, Adoption and Retention, and Task Success. This framework defines UX metrics of the product.

  • Happiness: This index measures user attitudes, often collected via survey. For example satisfaction, perceived ease of use, and net-promoter score.
  • Engagement: This index measures the level of user involvement with the product or feature, over some time period. Examples might include the number of visits per user per week or the number of photos uploaded per user per day.
  • Adoption: This index measures the number of new users of the product. For example the number of accounts created in the last seven days or the percentage of Gmail users who use labels.
  • Retention: This index measures the rate at which existing users are returning. For example: how many of the active users from a given time period are still present in some later time period?
  • Task success: this includes traditional behavioral metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate. This category is most applicable to areas of your product that are very task-focused, such as search or an upload flow.

As a PM, your metrics don’t have to hit all the different categories in the HEART framework. Instead you should choose the metrics that are the most important for your project. The HEART framework can help you decide whether to include or exclude a particular category. These can be applied at a number of levels — from the whole product to a specific feature.

How to map the HEART framework to product metrics.

How do we map the HEART framework for our product or feature? We do that in three steps

  1. Goals: What are the underlying business goals of the product or feature. This step is critical to deriving meaningful metrics. Goals also bring alignment among the stakeholders. When goals are formalized, essentially everyone has agreed on the reasons for releasing a product or feature.
  2. Signals: Signals help track the progress of the goal. Signals break down the high-level goal into lower-level steps. The metrics are selected based on the signals
  3. Metrics: Select the metrics for each signal. You would ideally select a metric for each category of the HEART framework. You can also prioritize the most important metric based on business or product goals.
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/15b46be8-759b-4c60-aeb7-6dee33c4dfff/Untitled.png

Credit: https://www.dtelepathy.com/ux-metrics/#quality

Lets apply the HEART framework to a real product and walkthrough the steps for identifying the metric, measuring them and listing the tradeoffs or limitations.

Interview Scenario: YouTube Metrics

Interviewer: Walks in the room. After the initial introductions, “ I would like to discuss about product metrics”

You: Sure! (Internally you are just hoping that the problem is a easy one :))

[Interviewer will discuss a specific feature or real product example and ask you to choose the most important metrics and how to measure them?]

Interviewer: Let’s start by assuming that you are a program manager working on YouTube. You are tasked with creating measurable metrics for YouTube, I would like to see which metrics would you choose.

You: OK. Here is what I would do

I will show you the solution by applying the HEART framework below. Note that this is not the only framework you can use. I have written a blog post earlier on using GAME framework to solve metric interview questions.

We have five categories for the HEART framework. We will select the Goal, Signal, and metric for each category and apply them to the YouTube product as a whole.

YouTube is a social video sharing platform that makes it easy to share and watch videos. YouTube has both free (ad-supported) and paid subscription model for revenue. In this model we are working with YouTube as a free service and looking at different metrics.

Generally, revenue is NOT a recommended goal. Companies that focus solely on revenue fail to consider that user experience is a more important health metric for the overall company. First and foremost, deliver an incredible and worthwhile user experience. Then, you can consider monetization metrics.

Not all categories of the HEART framework will be applicable to every use case or feature. In our case of YouTube we pick Engagement, Adoption and Retention categories for metric decisions.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/a3709d62-3868-4618-8ff2-edea24df0223/heart-framework.png
YouTube Metric Framework

Trade offs and Limitations of the metrics:

We’ve defined key metrics for the YouTube platform, based on the HEART framework. We talked about user engagement, new user adoption, and retaining existing users. Now, let’s think of some pitfalls and potential tradeoffs from these metrics to give a complete answer to the interviewer.

Comments are not necessarily a positive user metric. While commenting users are engaged users, they may be frustrated, offended, or disgusted with the content they are viewing. To mitigate this concern, it’d be helpful to use sentiment analysis tools for comments to check if these comments are generally positive or negative in nature.

Watch time might not be a positive metric if taken to the extreme. There are cases of user addition to YouTube viewing habits.

Summary:

Always use a framework to answer metric questions. Do not jump into solution space immediately.

In this blog, we used the HEART framework to answer the interview questions on YouTube metrics. I will show another example using the AARRR framework in the next blog post.

I have outlined a step-by-step method that you can use to solve metric questions. I encourage you to try solving a couple of the sample questions using the steps defined in this blog.

0 0 vote
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments