How Duolingo Kills the Product Management Game

I regularly get asked for case studies of companies doing product management well. There’s such an enormous amount of content about what “good” looks like, and yet there’s a dearth of case studies of companies implementing these methods successfully. When I come across such case studies I feel compelled to share them. The case study I’m sharing today is not from one of my clients. It’s one I read in another product management newsletter, Lenny’s Newsletter. The case study is from Duolingo. While I’m not going to reprint the piece here (for obvious reasons), I did want to recap some of the key themes that resonated with me.

The full Duolingo case study can be found here.

Focus and prioritization

The first thing that grabbed my attention was the Duolingo team’s decision to focus on retention of existing users. This is such a critical decision to make up front because it immediately provides a yes/no decision flow for any feature ideas. Does this idea help us drive retention? Yes, let’s consider. No, not now. It also helps the team defer any requests from across the company to acquire new users.

All too often I meet teams who are trying to do it all. In the end, they achieve none of their goals because they’re spread too thin. They lack the focus and discipline to dig deep and analyze why specific efforts aren’t having their desired results. By deciding which key result to focus on first (and by definition, NOT work on others), the team can put all their effort and brainpower into working on this specific problem.

External inspiration

In an effort to drive up retention, the team sought inspiration from other types of products. In their case it was casual gaming and Uber that offered the inspiration that drove their next set of hypotheses. Oftentimes we don’t seek any external inspiration. If we do venture outside of our own products, we tend to stay in our industries.

There’s so much to be learned from other industries. When I worked at TheLadders (what now feels like 1000 years ago), we often sought inspiration from online dating sites. In essence, we (a job board) had similar challenges to the dating sites: bringing together two parties seeking specific situations while not being aware of each other. Dating sites have tons of experience in this area. We were constantly learning from outside the job search industry to find inspiring ideas to solve our problems in unique ways.

Minor improvements are not “wins”

When they next set out to acquire new users, the Duolingo team’s efforts saw a 3% lift with their “free month” offer. This was well below their target. They saw this as a minor success and, rather than pivoting, iterated repeatedly down the same path. Growth was stagnant. My takeaway here is that just because the team saw some positive results, they weren’t strong enough to signal a successful hypothesis. Results don’t always have to be negative to signal a false hypothesis. In some cases, not enough positive impact is a good enough reason to kill the hypothesis and go down a new path. This is tough for teams to swallow because it feels like “we found something that works.” In reality, though, if the numbers don’t move high enough and fast enough, it’s a fail.

The other thing this illustrates is a team’s decision about what “good enough” means to them and to the company. When we set key results, we often wonder what the number should be. Understanding that a 3% lift isn’t good enough helps a team to change course sooner because they understand that this ROI isn’t worth the work they’re putting into this idea.

Data-informed understanding of what customers “do” is a competitive advantage

Working in close collaboration with engineering and data science, the VP of Product at Duolingo built a data-informed model of how users behave in their system. It wasn’t a simple DAU (daily active users) or MAU (monthly active users) model. Instead, the team built a set of segmented behavioral metrics that reflected exactly how “active” (or not) users of Duolingo were. They then modeled the relationships between these behaviors. For example, using this model they could answer questions like, “If we improved at-risk users by 1%, what will the impact be on DAU and MAU?” Once again the team found itself with the ability to determine exactly which lever to pull to maximize a return on their work. And, as before, prioritization became much easier given this level of insight.

Optimizing what you have can often be more impactful than building something new

In the case of two features, streaks and push notifications, the product team at Duolingo opted to optimize rather than reinvent. The features had a decent impact on user metrics, but the team felt like they could get more out of them. Working cautiously—not to negatively impact their current success—the team experimented and A/B tested their way to better versions of each of these features. The goal was to improve their traction while not ruining the user experience. Not having to create something from scratch meant the team had more runway (both budget and time) to work with since they were optimizing features that were already in the market.

Collaboration to move quickly

Perhaps my biggest takeaway from this case study was the constant theme of collaboration and shared understanding. In nearly all the examples shared the product team worked with engineering, data science and design to come up with hypotheses for product improvement. Data and research wasn’t outsourced to another department. Cross-functional worked together to build models, create hypotheses and run experiments. As they learned together, they were able to make decisions together, quickly, without a lengthy “running it up the chain” decision-making process. Finally, the team used data (both qualitative and quantitative) to make decisions, regardless of how they felt about their feature hypotheses.

In many ways, this is an excellent case study of how modern product management works in the wild. It showcases the key themes of humility, learning, agility and collaboration to generate evidence for the decisions the team is making. All of this was happening in the service of strategic objectives prioritized through user behavior. There’s a lot to learn and apply here.

What I've been up to

With Spring nearly upon us I’m frantically trying to get in a few more days of skiing before everything melts and we shift full force into mountain biking season. My other passion these days has been Padel. For those of you not familiar, it’s a combination of racket ball, squash and tennis (though definitely not pickleball). It’s social, easy to pick up and a ton of fun. I’ve been getting out to play matches as often as I can.

On the professional front, I’ve picked up a couple of public speaking gigs coming up. If you’ll be in Hamburg at the Product at Heart conference, you can see my keynote there as well as pick up one of the last remaining seats for my OKR workshop.

Watch, Listen, Read

Watch: Picard, Season 3 – What can I say, if it has Star Trek in the name, I’m in. Picard’s third and final season has started streaming and brings together all the great characters from Star Trek: The Next Generation. It’s been an action-packed kickoff to the season so far!

Listen: Rare 90’s Underground Hip Hop (playlist) – I can’t get enough of this playlist. I put it on as background music for writing (it’s on right now!) or any other time I need to get into a good flow. The title says it all.

Read: Sony and Honda’s EV goes where the Apple Car never did – This short article over at The Verge puts forward yet another case study that software has eaten the world. What I like about this story is that it moves us past the Tesla case studies and references and on to other organizations realizing that delivering value is continuous regardless of the product you’re selling.

Interested in working together? Please reach out. 
In case you need it, here's a description of what I do.