You are launching a messaging app. Define 2 metrics that you'd choose to monitor app performance during the first months. Why you chose them?

Answer:

The answer below is not just for messaging apps, but applies to pretty much any new product launch. Ultimately, you want as many users as possible using your product and you want the number of engaged users going up over time. An old business adage says that your product is either growing or dying. And you don’t want it to be dying.

Ideally, you always start by targeting growth as the high level goal, and then narrow it down to something more specific that should have an impact on growth. You typically need two components in order to have product growth:

• Ability to acquire new users
  
  
• Ability to retain current users
  
  

It is really hard to grow if you lose your past users and, certainly, not sustainable in the long run. It is pretty much impossible to grow if you don’t increase the user base. Therefore, you should pick two metrics that reflect the two bullet points above.

Regarding acquiring new users, a good metric could be: new sign ups per day from users who send at least 1 message within the first 2 days. Note that pretty much all the metrics related to new user acquisition follow a similar template:

• Define the key event you care about -> "sign-ups" here
  
  
• Define hard thresholds that work as quality guardrails -> "at least 1 message within the first 2 days", aka you care about high quality sign-ups only. You can identify the most appropriate thresholds via a decision tree
  
  

Regarding retention, pick the key action you want your users to perform. If possible, pick actions that incentivize other user actions. That will create a virtuous circle. For instance, here, we can choose average messages per day per user, or percentage of users sending at least X messages per day (in case you want a metric robust against outliers/power users). Later in the course, there is a specific retention-related challenge about how to estimate retention.

Btw engagement and retention are often two sides of the same coin. In the high majority of cases, retained users are the most engaged ones. So, when thinking about retention, you could start by defining engagement. In most cases, ads-companies (FB, Google, etc.) talk about engagement and e-commerce (Airbnb, Amazon, etc.) talk about retention, but this is more of a convention than anything else. The two concepts are so strictly related to be almost the same.

Avoid vanity metrics. That is, metrics that look good, but are useless. A good way to quickly identify if my metrics are useful is: imagine a bot is creating a bunch of fake accounts. Will my metrics go up? In the example above, we are good. The retention metric will drop and alert us that something not good is going on. On the other hand, if you ONLY looked at new sign ups per day, you would be happy with the fake accounts.

Finally, as the company grows, you will start narrowing down those key metrics, focusing on smaller metrics that are correlated with the larger ones. That is, the broad retention metric might become percentage of users with a profile picture, because you previously proved that the narrow metric is correlated with retention (if profile pictures go up, messages are also expected to go up). No matter what, it is going to be almost impossible to justify working on optimizing a given metric unless you can link it to one of the two key growth metrics above. If you cannot link it, no product manager will prioritize testing your product idea, you won't get test traffic for the A/B test, etc.

Many additional product questions and case studies are included in the full course in product data science.
Also, if interested in Gen AI DS case studies, you might want to check out the brand new Gen AI Data Science course.