A/B Testing: User Referral Program


User Referral Program



Goal


Almost all sites have a user referral program where you can invite new users to try a given product. Typically, after the new user completes a transaction, you get rewarded with a certain amount of money or credit to be used on the site.

The goal of this challenge is to analyze the data from a referral program and draw conclusions about its effectiveness.




Challenge Description


Company XYZ has started a new referral program on Oct, 31. Each user who refers a new user will get 10$ in credit when the new user buys something.

The program has been running for almost a month and the Growth Product Manager wants to know if it’s been successful. She is very excited because, since the referral program started, the company saw a spike in number of users and wants you to be able to give her some data she can show to her boss.

  • Can you estimate the impact the program had on the site?
  • Based on the data, what would you suggest to do as a next step?
  • The referral program wasn’t really tested in a rigorous way. It simply started on a given day for all users and you are drawing conclusions by looking at the data before and after the test started. What kinds of risks this approach presents? Can you think of a better way to test the program and measure its impact?



Data


We have just 1 table downloadable by clicking here.


The table is:


referral - provides information about each transaction that happens on the site and whether the user came from the referral program or not.

Columns:

  • user_id : the id of the user
  • date : date of the purchase
  • country : user country based on the ip address
  • money_spent : how much the item bought costs(USD)
  • is_referral : whether the user came from the referral program (1) or not (0)
  • device_id : Id of the device used to make the purchase



Complete and Continue