You Might Not Need a Metric
Startups love metrics, and rightfully so. But what has more value? Knowing your speed, or that you're heading towards a brick wall?
Choosing the right metrics for your project is critical.
It guides your decision-making by giving you indicators of progress (or the absence of it), saving you from years of going fast in the wrong direction. Vanity metrics are everywhere. Feeling good about building the wrong product on time and on budget is one of them.
The startup ecosystem is filled with valuable information on how teams use data science to grow their products. Heck, we came up with Product-Led Growth not too long ago. Meanwhile, tracking metrics has become increasingly cheap, leading many founders to collect them like Pokémon. Chances are, only a few are relevant for your business, the rest only leading you astray. Best case scenario, you lose track of the big picture. Worst case, you start using them to tell the story you want to hear.
Defining which metrics to track is challenging, the answer being as unsatisfying as it is boring: it depends. There are no silver bullet solutions, no one-size-fits-all frameworks. I’m not here to make yet another list of “the metrics you must track, otherwise you’ll die”, to give you FOMO and drive some clicks. If that’s what you’re after, just type “startup metrics” into your favorite search engine, and voilà ! You can now spend hours listing all the things you cannot afford not to track.
Instead, my point is that tracking numbers for the wrong reasons will harm you more than tracking nothing at all. Ask what you should not be tracking. To help with that, you can assess metrics with the sanity checklist below.
- What question are you trying to address? What specific behavior or outcome are you trying to track? Why?
- Is the metric aligned with organizational goals? If so, how?
- Do you have enough data for the metric to make sense, statistically speaking?
- Is the question actionable? Meaning, is it likely someone will take action based on the numbers you’ll see?
- Who will be in charge of acting upon those numbers, and are they part of the conversation?
- What form of data can convince the decision maker(s)? Quantitative data, such as logs and metrics? Qualitative, such as surveys and interviews? Both?
- How much data is needed to convince the stakeholder(s)? And is this amount within reach?
- What will you do with a negative or inconclusive result? If it’s not going to change your course of action, why bother?
- How frequently will this metric be reviewed or updated?
Take for instance customer churn rate. The question this metric is answering is: Are customers continuing to use our product? But if you have only 10 users, it doesn’t make sense to track it, at least not in a quantitative way, since it’s not enough of a sample size.
Next, take the yearly number of lines of code shipped per engineer. Some organizations did try tracking progress that way in the last decade, using it as a heuristic of engineering productivity. Is this aligned with organizational goals? Well, if your goal is to ship more code, and you’re able to derive economic value from it, great. Otherwise, not so much.
At <FAANG mega-corp>, I used to work on an internal team which basically did nothing important, certainly nothing users ever think about.
The problem that we were trying to solve, well, we had already solved it about 3 years ago, yet the team was still expanding, and our product managers kept spitting out new (mostly useless) project ideas.
One of the main areas of focus was metrics. Leadership was obsessed with metrics, we measured everything. Not just plain empirical metrics (like # clicks), but also very complex metrics, theoretical models that we had data scientists work on, etc. At some point maybe 25% of the team worked on metrics related stuff, it was crazy.
Why were they doing this? Was it only to keep us busy, so that the team can continue to grow? Or maybe they were desperately trying to find metrics by which they could prove the team was actually still doing meaningful work? I don't know.
At some point I left, and moved on to a much smaller team. This team was working on a brand new product, one that customers actually pay money for!
One of the first things I said to my new product manager was "so, what metrics do you care about? how do you measure success?".
He was taken aback by my question. He seemed genuinely perplexed. He paused for a few seconds, then said "metrics? What do you mean by 'metrics'? I look at revenue, when the line goes up I'm happy".
I thought it was... insightful.
Taking the time to answer those questions will save you a lot of time and headaches down the road. I’ve seen too many founders get all tangled up in metrics, following them religiously because they read someone successfully used them to grow, yet not knowing what to do with them, apart from feeling compelled to make them go up.
I know, because I’ve been there. The choice overload is real, and your escape route is a sanity checklist. Feel free to steal this one.
Resources :
- Improving Engineering Productivity at Scale, a talk by Ciera Jaspan, Engineering Productivity Research group Lead at Google
- Avoiding False Metrics article on Seth Godin’s blog
- Lean Analytics book