How to think about retention?
In this article we'll explore the pitfalls and biases around retention rate and share a real case of boosting monthly retention by 30%.
Hi and welcome to the Corporate Waters weekly newsletter 🙌
I’m Mikhail and I'm excited to share my learnings to help you navigate the complex waters of product management, leadership, and corporate dynamics. Subscribe to unlock the full value.
Have questions, feedback or want to suggest a topic - please reach out.
Product reflections (Free corner). How to think about retention?
Real case (Paid corner). How we’ve accelerated retention by 30% at OLX.
Retention is a fancy product metric that’s thrown around frequently. Every product team I’ve worked with in the past 10 years has quoted and mentioned retention more frequently than any other metric. Whether using looker, tableau, amplitude or internal data tooling - if I just typed in “retention” in the search bar, I’ve always stumbled upon dozens of boards tracking retention in every possible way.
It’s particularly favoured by Data Analysts as you can slice and dice it from a variety of angles. You can break it by cohorts, first-use date, pick a certain “use case” as an activation metric (e.g. first use case retention) or assign a dollar value to it (e.g. net revenue retention). The opportunities are literally endless.
Moreover, stakeholders love it. It’s a business darling in close proximity to the P&L that could be easily translated into LTVs or GMV uplifts.
I’ve personally battled with retention improvement biases. What are the industry benchmarks? How can we improve our numbers to get closer or even exceed the competition? After years of fruitless experimentation, I have come to realize that retention in itself can never be the goal.
Framework for thinking about retention
Before we dive into the specifics, the basic framework you and your teams should stick to in order to avoid common “retention pitfalls”:
😎 Keep it simple
📶 Define your retention plateau in the physical world
🎯 Agree on the activation event
🥅 Don’t make retention improvement your goal
What’s wrong with retention measurements?
First of all, retention is a very high-level output indicator. It’s a representation of the holistic value behind your product and is rarely a helpful metric for opportunity discovery. If your retention is low, you will most likely know it even before looking at the actual retention rate numbers.
Secondly, retention can be easily misinterpreted. The biggest problem I have seen among teams is the arbitrary selection of an activation event. One team was bragging about a boost in monthly retention while another team was observing a decrease while measuring the same product. Yes, this is a possible scenario because one team selected "opening the app" as an activation event, while the other team was more diligent and focused on the "buy button" in the later stages of the funnel.
Thirdly, in the pursuit of perfect retention measurement, we sometimes forget that there are real people behind it. It's dead simple. No matter how much you push and try to improve your Net Revenue Retention for a core cohort of users, if there's no need for your product in the real world, it's not going to happen.
The basic structure of retention
I am always surprised with how such a simple concept as retention can be made so unnecessarily complex. I remember opening a tableau board with rolling retention split by twenty cohorts with indecipherable names and asking myself “How on earth did we get there?”. How could a simple junior school math formula be stretched to the point where the meaning behind it is lost?
👉 A quick primer on retention rate.
The most basic formula consists of three variables:
Activation event (this has to be very clearly communicated and agreed with across the whole team)
Timeframe
User base
You can add a bunch of bells and whistles to your retention formula and fog the minds of stakeholders, but I doubt it would be useful. In practice, I haven’t seen a single use case (besides customer acquisition and campaigns in general) where added retention complexity aided decision-making.
Start with the retention plateau
Forget about industry benchmarks. Start thinking about the user. How often does the event occur in real life? If you're selling fridges, don't expect the user to return more than once every few years. On the other hand, the needs to commute, eat, get dopamine, and communicate with other people happen on a daily basis.
That's where the retention plateau concept comes into play. It's essential to be mindful of it as early as possible in the ideation of your business. In terms of retention, a barbershop might be a great business because the need to get a haircut arises on average once a month. Therefore, once a month is the plateau that cannot be easily shifted. Men's hair isn't going to grow faster, no matter how many push notifications you send out.
Retention isn’t sensitive
From 2015 to 2023 onwards, I witnessed 600+ AB tests with products of varying maturity (from dozens of K to 80m+ monthly) and markets. None were able to move retention, even when sliced by cohorts. It's not sensitive enough to change. Longer periods of time are needed to trace the impact (1m+), but rarely do teams have the luxury of maintaining a running AB test for such extended timeframes.
If you ardently believe that the test will move the retention needle, the best way is to find a proxy that is sensitive enough (e.g., sessions per user or the number of active users) to trace the impact within a shorter period.
Why retention shouldn’t be your goal
Since it's a high-level output metric, product teams shouldn't commit to shifting it. There's a likelihood that your team doesn't fully understand the levers behind it, and even if it does, there is no clear horizon of impact that you can aim for.
My take is that retention and its proxies shouldn't be the goal or the hypothesis of your AB test, but rather a spill-over effect. On top of that, I'm a big believer that if what you're doing adds value to the user, the cumulative impact will be seen in the retention uplifts.
You don’t need to research retention
If you're targeting retention as an object of research, there's something really off with your team's discovery. As I've mentioned above, retention is an indicator of value. Hence, an understanding of your user, their journey map, pains, and needs should be the pillars that drive your research.
The teams that are overly focused on trying to understand the reasons for insufficient retention are most likely lacking a continuous discovery process and are fixated on the "retention as a framework" rather than focusing on "retention as a side-effect of value".