Say no to SaaS vanity metrics


“How’s it going?” is the start of many founder/investor interactions. After bouncing back with an automatic “Great!”, I’ve both said and heard many topline vanity metrics. Although most startups preach the mantra of transparency, it’s hard to keep the party talk from influencing your internal analyses. This proves problematic when a lot of commonly discussed metrics can hide the reality of a company’s performance. We all prefer rosy pictures, but focusing on the wrong numbers might convince you to stay the course when a change in strategy is needed. In SaaS businesses this can be especially dangerous, as downloads and signups are the first step in a long funnel to recurring revenue.

Monthly engagement v. total engagement

In general, using totals is a bad way to track progress and especially growth, even though these will be the numbers you likely use in public. Totals only go up, so it’s difficult to see all but the biggest changes. To more easily see changes and to track growth in the app’s usage, break down your overall engagement metric (for our company, it’s messages and voice minutes) into monthly totals.

Here’s an example: Suppose Acme Corp. has a messaging app (like Whatsapp or Line) that launched in January 2013. If you look at their Total Messages graph for the year, below, it only seems to tail off around October. However, if you look at the Monthly Messages graph, you can see the Acme app has seen declining engagement since July, and by October, monthly messages have halved. If the company is focused on total messages, it would be easy for them to miss the problem, meaning they’re unlikely to make the changes needed and reverse the decline.

Fig 2 Monthly messages

Monthly recurring revenue v. revenue

Everyone in a SaaS business knows Monthly Recurring Revenue (MRR) is the lifeblood of the business (with churn as the enemy). However, it’s not always the easiest value to accurately calculate once you factor in nuances like annual plans, custom plans and different payment sources. At SendHub, this meant we focused on total revenue for the month, which in turn changed the behavior of our sales team.

In focusing on total monthly revenue, the sales team was incentivized to sell annual plans, rather than monthly plans. This was great for reducing near-term churn, but given that annual plans are 10 months for 12, it reduces the MRR for the customer by 17 percent. The churn advantage makes this a very equitable exchange, however in tougher months we were tempted to offer discounts on the annual plans to top up revenue and thus take further reductions in MRR. This practice is essentially borrowing from future months, as you won’t get any additional money from those annual customers, making it even harder to hit your revenue goals in the following months.

As a startup, you also feel compelled to take money in any way you can. For SendHub that translated into taking money via means other than credit card — check, PayPal, wire transfer, in-app purchase, whatever the customer wanted. This made it hard to track in a centralized system and created a gap between our predicted MRR for a given month and the actual money we’d expect. The team stopped trusting the expected MRR value from credit cards, placing further emphasis on the total revenue number.

If you’re in a SaaS business you must have an accurate calculation of your MRR, even if it’s difficult, and align your sales team’s interests with growing MRR, at the expense of everything except churn. If you find yourself being pushed towards short-term wins at the expense of MRR, I suggest taking the hit now and focusing on the weakest areas of your customer funnel. In six months you’ll be grateful for the decision.

Monthly active users v. ignoring new users

Over the last 15 years, the tech industry has evolved from focusing on hits, visits, downloads and users, and more recently consumer apps have turned to monthly active users (MAU). While SendHub was in Y Combinator’s winter 2012 batch, we were pushed to focus on a slight variation of MAU, which excludes all users who signed up in the last 72 hours. We call it highly engaged users (HEU). Why is this metric better? Because these engaged users cared enough to come back and use our service a reasonable amount of time after they signed up.

The difference between MAU and HEU can be huge. At SendHub we expect just 55 percent of new users to return after three days, so although MAU is certainly a better user base metric than visits or downloads, it can contain significant padding and is unlikely to be an accurate indicator of those who are actually using the app. MAU can be especially dangerous when you get a sudden rush of traffic, like from press or getting your app featured. Many of the users acquired by these means are not more than vaguely interested in the product and will touch it once, never to return. However, this traffic will cause an instant spike in your MAU, which will then nosedive in 30 days as those “one use” users disappear from the end of the activity window. If you’re looking at HEUs instead, the spike will be delayed and only include those new users who were genuinely interested.

In an ideal world, you’ll be able to break out cohorts from within your HEU number, like signup source and date. This will tell you about how the changes you ship influence user engagement and also which of your clients (iOS/Android/Web) is performing the best. Armed with this data, you’re in a much better position to make acquisition decisions back at the top of the funnel.

Invest in better analytics

The success of your company relies on finding your customer’s pain point and solving it effectively. To do that, you need to know what your users are doing. While the simple, and accepted, metrics offer a reasonable indication of success, they are susceptible to significant skew, and it’s hard to stop that train when it feels like success. It gets even harder when these more accurate metrics require engineering effort to build and maintain. Ultimately, it’s a difficult balance to strike, but while we’ve regretted shipping many features at SendHub, we’ve yet to regret shipping any analytics.

Ash Rust is the Cofounder and CEO at SendHub. You can follow him on Twitter: @AshRust


Comments have been disabled for this post