METRICS: Conversion rate down, but absolute number of conversions up. Is it good or bad?

This question is a simpler version of the classical question: “A given metric dropped, how would you figure out why?” (how to answer that question is here). However, it is simpler since in this question you don’t have to figure out whether a metric dropped because the denominator spiked or the numerator dropped. That’s already clearly stated in the question: # of conversions is up, but # of users is up proportionally more, leading to a drop in conversion rate. You can jump directly to trying to understand which segment(s) are causing this change.

Firstly, for that to happen it means the site has started attracting new users who convert at a lower rate than the site average conversion rate. If conversion rate is 3%, and the site is getting a sudden spike in visits but these new visits have 2% conversion rate, eventually there will be a decrease in overall conversion rate, but an increase in absolute number of conversions.

This is actually a really common situation. Indeed, most marketing events will lead to this outcome. Blasting an email to all past users will likely lead to a spike in visits, and increase in absolute number of conversions, but a drop in conversion rate. The mobile app suddenly being promoted by Apple/Google app store will also lead to the same outcome as well as any major advertising campaigns.

Figuring out the reason is pretty straightforward. As usual, it boils down to comparing segments before the change vs after the change and figuring out the user segment that spiked. In the high majority of cases, here it will depend on the variables user_source (i.e. how did users come to the site: SEO, ads, email, etc.) and/or user_location, since often these spikes are in a specific location (i.e. Apple promotes the app in a specific country app store).

Whether this change is good or bad is the key part of the question. It obviously depends. If absolute numbers went up by a really small number and the spike in visits came from a super expensive ad campaign, then it is not good. But leaving aside extreme situations, in most cases this change is good.

It is good for two reasons: the most obvious one is that the company is making more money. But also because it is typically much harder to bring people to a site vs to make them convert. Increasing site visits presents a lot of opportunities for long term revenue gains. The lower conversion rate for the new visitors is not a fixed number that will never change. Data scientists can start looking at these new data and come up with product ideas to increase conversion rate. After all, this is exactly where data science excels: identifying the worst performing user segments, developing hypotheses on why that’s happening, and designing A/B tests to increase the target metric.

Note how this question touches a super important topic present in many other questions: even for the most important metrics, there can be situations in which that metric going down can be good for the business. The opposite is also true. If conversion rate spiked, but absolute number of conversions is down, that’s not good. After all, you could easily achieve that by, for instance, blocking access to your site from all countries except for your best performing one. But no one would ever do that. Especially if a metric is a proportion, checking if that metric is up or down is often not enough. You also want to check how denominator and numerator taken independently were affected.

Complete and Continue