10 Comments

Summary:

It’s easy to get caught up in vanity metrics: the numbers that make you look good even when your startup is stumbling. But here’s why you should examine those oft-quoted stats more closely, and the metrics you should consider swapping them for.

Mirror

“How’s it going?” is the start of many founder/investor interactions. After bouncing back with an automatic “Great!”, I’ve both said and heard many topline vanity metrics. Although most startups preach the mantra of transparency, it’s hard to keep the party talk from influencing your internal analyses. This proves problematic when a lot of commonly discussed metrics can hide the reality of a company’s performance. We all prefer rosy pictures, but focusing on the wrong numbers might convince you to stay the course when a change in strategy is needed. In SaaS businesses this can be especially dangerous, as downloads and signups are the first step in a long funnel to recurring revenue.

Monthly engagement v. total engagement

In general, using totals is a bad way to track progress and especially growth, even though these will be the numbers you likely use in public. Totals only go up, so it’s difficult to see all but the biggest changes. To more easily see changes and to track growth in the app’s usage, break down your overall engagement metric (for our company, it’s messages and voice minutes) into monthly totals.

Here’s an example: Suppose Acme Corp. has a messaging app (like Whatsapp or Line) that launched in January 2013. If you look at their Total Messages graph for the year, below, it only seems to tail off around October. However, if you look at the Monthly Messages graph, you can see the Acme app has seen declining engagement since July, and by October, monthly messages have halved. If the company is focused on total messages, it would be easy for them to miss the problem, meaning they’re unlikely to make the changes needed and reverse the decline.

Fig 2 Monthly messages

Monthly recurring revenue v. revenue

Everyone in a SaaS business knows Monthly Recurring Revenue (MRR) is the lifeblood of the business (with churn as the enemy). However, it’s not always the easiest value to accurately calculate once you factor in nuances like annual plans, custom plans and different payment sources. At SendHub, this meant we focused on total revenue for the month, which in turn changed the behavior of our sales team.

In focusing on total monthly revenue, the sales team was incentivized to sell annual plans, rather than monthly plans. This was great for reducing near-term churn, but given that annual plans are 10 months for 12, it reduces the MRR for the customer by 17 percent. The churn advantage makes this a very equitable exchange, however in tougher months we were tempted to offer discounts on the annual plans to top up revenue and thus take further reductions in MRR. This practice is essentially borrowing from future months, as you won’t get any additional money from those annual customers, making it even harder to hit your revenue goals in the following months.

As a startup, you also feel compelled to take money in any way you can. For SendHub that translated into taking money via means other than credit card — check, PayPal, wire transfer, in-app purchase, whatever the customer wanted. This made it hard to track in a centralized system and created a gap between our predicted MRR for a given month and the actual money we’d expect. The team stopped trusting the expected MRR value from credit cards, placing further emphasis on the total revenue number.

If you’re in a SaaS business you must have an accurate calculation of your MRR, even if it’s difficult, and align your sales team’s interests with growing MRR, at the expense of everything except churn. If you find yourself being pushed towards short-term wins at the expense of MRR, I suggest taking the hit now and focusing on the weakest areas of your customer funnel. In six months you’ll be grateful for the decision.

Monthly active users v. ignoring new users

Over the last 15 years, the tech industry has evolved from focusing on hits, visits, downloads and users, and more recently consumer apps have turned to monthly active users (MAU). While SendHub was in Y Combinator’s winter 2012 batch, we were pushed to focus on a slight variation of MAU, which excludes all users who signed up in the last 72 hours. We call it highly engaged users (HEU). Why is this metric better? Because these engaged users cared enough to come back and use our service a reasonable amount of time after they signed up.

The difference between MAU and HEU can be huge. At SendHub we expect just 55 percent of new users to return after three days, so although MAU is certainly a better user base metric than visits or downloads, it can contain significant padding and is unlikely to be an accurate indicator of those who are actually using the app. MAU can be especially dangerous when you get a sudden rush of traffic, like from press or getting your app featured. Many of the users acquired by these means are not more than vaguely interested in the product and will touch it once, never to return. However, this traffic will cause an instant spike in your MAU, which will then nosedive in 30 days as those “one use” users disappear from the end of the activity window. If you’re looking at HEUs instead, the spike will be delayed and only include those new users who were genuinely interested.

In an ideal world, you’ll be able to break out cohorts from within your HEU number, like signup source and date. This will tell you about how the changes you ship influence user engagement and also which of your clients (iOS/Android/Web) is performing the best. Armed with this data, you’re in a much better position to make acquisition decisions back at the top of the funnel.

Invest in better analytics

The success of your company relies on finding your customer’s pain point and solving it effectively. To do that, you need to know what your users are doing. While the simple, and accepted, metrics offer a reasonable indication of success, they are susceptible to significant skew, and it’s hard to stop that train when it feels like success. It gets even harder when these more accurate metrics require engineering effort to build and maintain. Ultimately, it’s a difficult balance to strike, but while we’ve regretted shipping many features at SendHub, we’ve yet to regret shipping any analytics.

Ash Rust is the Cofounder and CEO at SendHub. You can follow him on Twitter: @AshRust

You’re subscribed! If you like, you can update your settings

  1. Hi Ash, nice article.
    Agree with the focus on metrics.
    But in the earlier days how did you balance engineering time on product vs building better metrics?

    It’s an area we struggle with at the moment and your thoughts would be great.
    Jindou

    1. We frankly did not do a great job at this during the early days. We took a measure everything approach via Mixpanel events, which ultimately turned into data overload and is something Mixpanel’s onboarding warns against. I’d suggest spending some time on the 3 key metrics that really indicate success and build just those. That will, hopefully, allow you to focus on the product, without flying blind.

  2. Very good and timely article. Your HEU (Highly Engaged Users) metric is certainly a key insight that can make your business metrics more realistic and thus more powerful over time.

  3. Excellent insight, this HEU metric that you use at Sendhub. Much more meaningful than MAU or equivalents, as you point out.

  4. Every bit of advice here is good, but there’s one little caveat IMO that needs to be said, you shouldn’t personally get caught up in your own vanity metrics, but I think understanding the concept of social proof and why it’s important and how businesses use it to impress people is pretty vital. http://www.facebooklikesreviews.com/articles/what-is-social-proof

    Some of those vanity metrics can still be useful in terms of how you portray yourself to people, in some respects.

    1. Great point. Social proof is an essential part of the customer funnel and a large topline number can help, although I feel like recognizable logos is the more common approach.

  5. Great article and agree with the focus on truly insightful metrics. What tools do you use to track these metrics at SendHub?

    1. We started off and still use mixpanel, although we’ve found it hard to do more complex metrics like HEU in that software. Now we use our own db for storage then leftronic and periscope.io for visualization.

  6. Couldn’t agree more. We build products for startups and found no low cost solution to let us just build in a simple solution for measuring growth, retention, etc.

    So we built our own. It’s free and does the job, see http://www.saasrunner.com It takes about an hour to integrate.

  7. Denis Duvauchelle Wednesday, March 26, 2014

    Nice piece! It can be really easy to get sucked into believing that everything is going well, when in fact it’s falling apart. As Ash Maurya stated, your metrics should look like something you are not proud of. It’s also important to gather the right info from as early as possible. We put a simple intro guide to metrics here for anyone who needs some guidance on how to get started: http://blog.twoodo.com/288/best-intro-guide-to-saas-startup-metrics/?utm_source=blog&utm_medium=comment&utm_campaign=techblog

Comments have been disabled for this post