As a PM, you need the skill of data analysis. When?

Part 2 of 3 of the series: Develop the critical PM skill of data analysis

In the previous issue I explained why it’s important for PMs to know the skill of data analysis. But under what circumstances should you use this important skill? When should you use it? Turns out, all the time. These are the most frequent scenarios.

Supporting one-off customer requests

For a mature product in a large company with thousands of users, customer requests are typically handled by a dedicated support team. For smaller companies and less mature products, as a PM you may be playing a support role yourself. In any case, you you may find yourself just like Sherlock Holmes, investigating a customer-reported problem. Especially when admin tooling isn’t available yet, you need to dig into the customer’s data, trying to determine the root cause of the issue and reliably reproduce the problem. You also want to understand the impact of the problem in order to prioritize a fix. Is a weird bug preventing a handful of customers from downloading 5 year old PDF statements from your finance app? Or is a major UX flaw preventing 60% of users from getting new loans, your core business offering? Answering these questions require data analysis. 

If you think an engineer can and should be doing this investigation, that’s only partially true. You want engineers focused on doing hard technical work and advancing your planned roadmap. So you yourself should do that initial triage with your data analysis skills to understand the impact. Only if you deem the issue high priority enough, then bring it to your engineering team for help. In fact, don’t assume an engineer would be much better than you in finding the root cause of a problem. Oftentimes software products are quickly cobbled together with poor documentation by different engineers, especially in fast-growing companies.

Major production outage or widespread customer problem

When there’s a major product outage or a widely reported customer problem, in many ways it’s not so different compared to the previous scenario of addressing a single customer’s concern. You still dig into the data and investigate the root cause. In this case though, the impact is clear. It’s an urgent issue and likely many folks are working with you. So you have an additional task of coordinating communications. But that’s for a future newsletter issue to discuss. 

Tracking product usage and feature adoption

Whenever you hear on the news that Facebook has over 2 billion monthly active users (MAUs), know there’s a PM who’s ultimately responsible for tracking that number and trying to push it even higher. In the world of product, defining, tracking, and improving usage is super important. The day-to-day PM activities ultimately serve to achieve this principal goal. In particular, users consistently return to your product if there are features that they find valuable. So per a first-order simplifying assumption, feature adoption drives product usage, and product usage drives business success. That’s why PMs care so much about usage. It’s a universal standard currency. And the only way to truly understand and improve usage as a PM is to collect and analyze usage data yourself, or at least be a key stakeholder and driver in that process. 

If you’re working as a PM in a medium-sized company or even in an early stage startup, there’s usually already an assumed business model. (This is very much the case in a large company.) You’re tasked to improve usage without much regard to finances. Finances and unit economics are important for product folks to understand, but less so early on in your career. Worry about usage first in your product. The dollars and cents will follow.

Tracking product health metrics

Beyond basic product usage metrics, you should have a series of product health metrics to monitor. In particular, you don’t want to sacrifice product health on the altar of usage. For example, you could juice product usage by heavily investing in marketing, getting new users to sign up for your product. But if your product is fundamentally flawed, users will quickly leave after discovering the poor experience. In this case, user retention is the health metric to monitor. It’s a strong indicator of product health. Users should find long-term value in your product and continually return to use it, especially if the cost of initial acquisition is high. Other health metrics include acquisition and onboarding times, NPS (net promoter score), and how often and how quickly users are performing certain tasks specific to your product. All these require a deep understanding of data analysis and practical know-how of collecting and tracking that data.

Interestingly, often other departments in medium or large companies care more about health metrics over north star usage metrics, since it impacts their day-to-day more. For example, marketing folks are responsible for generating a high number of quality leads. So they care very much about acquisition and onboarding metrics to see their efforts be recognized. Support folks care about retention. They want to keep as many users as possible, often by offering manual human support, even though users may be leaving due to a fundamental product problem.

Analyzing users and feature correlations

Customer development is the process whereby you develop an understanding of your product’s customers, whether existing or future. It’s really the unstated goal behind product development, and the difference in terminology is on the emphasis. Do you start with the product, or do you start with the person using the product? Especially for early-stage startups or new product initiatives in an established company, you should be focused on the user first. The product should change to adapt to the needs of the user. And so a major part of that is analyzing data collected from users, and even setting up the systems and experiments to collect that data in the first place. That could be user surveys, in-person user interviews, or usability studies. It’s critical to design these methods of analyzing people in a way that is amenable to hard data analysis after the fact.

As your product grows and you gain more users, they will start using it in ways you didn’t anticipate. Especially as the complexity of your product grows, how do you map out its future direction? So it’s important to understand how your users are using your product. Is the usage of one set of functionality highly correlated with the usage of another set? Why? Do you want to change and influence this behavior? This type of product development definitely requires more advanced data analysis, and oftentimes is the most valuable, especially as your dataset grows.

Caveat: Product development is not a science

As a final word, it’s important to note that product development is not a science, and thus you shouldn’t treat your data analysis as a science either. In traditional academia and research, you are studying a phenomenon. Your goal is fundamental understanding. This is science. You are doing your best to not interfere with a system, even as you measure and study it. The results of your work may have downstream implications, but that’s not part of your immediate research agenda. Think of hard science fields like physics or biology. Your goal is not to change what you are studying. In stark contrast, consider developing products. Even though you are studying your users as a phenomenon (customer development), you are very much interfering with them. You may be creating a new product to help solve a particular problem. And as you observe them use your product, you will change that product over time. The product becomes a vehicle of interference, and later on as the product matures, becomes part of the system you are observing itself.

All this is critical to understand because traditional data science is focused on studying large datasets. Even in product companies, you have research teams (data scientists) looking at user behavior at internet scales. But as a PM, most of the time your immediate goal is to make timely product decisions. You may have imperfect data. But that’s okay. Because the existing system source of truth is constantly evolving with each product change anyways. Furthermore, there’s other dynamics out of your control like the marketing department suddenly widening the top of the funnel to get more users. Even if they coordinate with you ahead of time, it’s very difficult to try to account for that in your data analysis. So a key principle for data analysis for PMs is to make sensible simplifying assumptions. And that’s truly based on gut and experience. Again, since product development is not a science, it’s not repeatable! You can’t reproduce the conditions and perform the same experiment again to strengthen your earlier results. The market, your users, the product, and even your competitors are constantly changing. Embrace this uncertainty and do the best that you can.

How to analyze data

In the next and final issue of this series, Develop the critical PM skill of data analysis, I’ll show you how to analyze data with specific techniques, like database SQL. Visit the website to see past issues.

Thanks for reading! 🙏

Mentorship in your inbox: Subscribe to Product Management 101

Product Management 101 covers practical tips and basic skills for aspiring and new product managers.

I’m Victor Wu, Head of Product at Tilt Dev. I’ve been creating digital products for over 15 years, and mentoring folks for much of that time. Subscribe, and reply to email updates with questions you’d ask in a real-life mentoring session. I’ll answer them in future issues.