24 Comments

Summary:

In 1964, Justice Potter Stewart, who was having difficulty explaining what exactly he meant by “hard-core” pornography, famously said, “I shall not today attempt further to define the kinds of material I understand to be embraced…but I know it when I see it.” When entrepreneurs ask […]

lock picking

In 1964, Justice Potter Stewart, who was having difficulty explaining what exactly he meant by “hard-core” pornography, famously said, “I shall not today attempt further to define the kinds of material I understand to be embraced…but I know it when I see it.” When entrepreneurs ask investors what they’re looking for in a startup, the response they get is often something along the lines of “traction.” And when asked to describe traction, most investors channel Justice Stewart, saying only, “I know it when I see it.”

Investors seek to take on the risks they can control and minimize the rest. The biggest unknown variable in the risk equation is usually the market, as in, “Is there even a market for your product?” This is especially challenging when it comes to products aimed at consumers, as you cannot possibly speak with several million people to get their feedback on a product before it’s even launched. So investors want to see enough usage of your product to give them confidence that there is both a big enough market to generate a large exit, and that you have nailed the product/market fit. That is what we mean by traction.

Misunderstanding the objective of traction, a number of Silicon Valley entrepreneurs have mastered the art of driving millions of people to a site or application regardless of whether the product solves a real problem or not. At best, this approach skews the data used to discover if a product is getting the kind of usage that would indicate there’s a big enough market for it. At worst, it’s spam.

Hackers hack everything, not just code

That things would go down this path should have been obvious. I’m not sure who kicked off this hacking traction phenomenon, but a small group of extremely bright entrepreneurs now use a very sophisticated form of multivariate testing. For example, they simultaneously test an extremely large number of variables on a registration page in order to maximize registration (the copy, size, color, font and placement of data fields; the format and text of buttons; the flow of pages), then do the same thing on the “invite friends” page (the text of the body and subject of the invite email), and so on.

This method takes advantage of latent human behaviors to drive registrations and visits — and in terms of those goals, it’s unbelievably effective. While this knowledge is still relatively well guarded, it is slowly making its way through Silicon Valley.

The insight that other pages say more about the pages they point to than such pages say about themselves (PageRank) was a revolutionary idea that led to Google Search. Machine learning has done a great deal to tune search, but machine learning without PageRank would not have resulted in Google.

Similarly, using very sophisticated optimization routines to maximize desired behaviors in a social application is a great way to make a good thing better. However, using these techniques too early can muddy the waters so that you don’t know if what you have is any good or not. If you were going down the wrong path, wouldn’t you want to know that as soon as possible?

Now that these entrepreneurs are returning to investors with “traction,” some are having trouble raising money. They are, understandably, confused. “You asked for traction — here’s your traction. Show me the money!

Too much of a good thing

Multivariate testing and other optimization schemes can be a great way to make a good product even better, and they are underutilized by many companies. But too many startups have begun misusing such traction techniques as a strategy rather than as a tactic, inadvertently destroying the feedback needed to build a great product.

As for me, I shall not today attempt further to define the kinds of businesses I hope to invest in…but I know them when I see them.

Mike Speiser is a Managing Director at Sutter Hill Ventures. His thoughts on technology, economics and entrepreneurship will appear at this time every week.

  1. UNBELIEVABLY great post. Why hasn’t this been discussed before? “Traction” as page views or free signups are such ethereal measuring sticks when you understand the underlying tech. So…you paid a Russian click farm, got the best PR firm. Or tweaked your signup page to spam and confuse. And, ended up selling your VC’s the Brooklyn Bridge. Awesome. For the win.

    We know all these techniques (and more)…but chose to stick with would would bring us boring, but real, results. For example, MySpace makes multiple pages out of systems that could be one. Why? Drives page views and ads. On the other hand, it annoys users, makes it spammy, and, important: tweaks page view perception. We chose to go the other direction…tortoise vs/ hare…and make the interface for the user, and the site for real usefulness. For example, people add hundreds of items on our site from the homepage (so, obviously yeah, our pageview numbers are lousy – big deal; we have lower IT costs too).

    imo, these are things that should be praised on the blogs. Quality sites, not quantity sites. Quality can’t be faked.

    Again, GREAT post…sorry for the rant.

    Share
    1. Thank you for the kind words.

      The market ultimately [usually] rewards quality – witness the numerous 10MM, 20MM, and 50MM uniques per month applications/sites that cannot earn enough revenue to turn a profit. Sometimes the press and investors reward the quantity camp early on, but consumers reward companies with great products over the long-term.

      Share
    2. @iTrack Hey I agree with what you said and went and checked out your unique COOL site.

      As far as the topic the internet (world) has always been poluted with scammers I just don’t know how you can wake up everyday knowing you scam people :( Anyway people that all of a sudden are gurus and experts at social media I have found aren’t to social and in fact in real LIFE they are mostly GOOFS.
      We will hear soon how someone came to their house and get their REFUND.
      The whole reality of success in America is often sadly the bigger the Douche and arrogant you are the more success and people do for you > looking up to these people I don’t get it :)
      Cool post stumbled

      Share
  2. Well said.
    The “social gaming” phenomenon seems to be built on this premise of optimization to build addicting websites .. errr … social games. In designing games, you choose features that you think users will enjoy. Many social gaming companies design by choosing features that increase ARPU or a similar metric. Both processes result in addicting games, but it’s like involuntary manslaughter versus murder. The end result may be the same, but the intent is different. And intent makes all the difference in the world.

    It was when I read the ‘Startup Metrics ffor Pirates’ presentation that it clicked for me: http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version

    Share
    1. I really enjoy Dave’s “Pirates” thinking. Dave also runs a conference called Startonomics (http://startonomics.com/).

      I would also encourage you to check out KISSmetrics (http://kissmetrics.com/) and Eric Ries’ blog (http://startuplessonslearned.blogspot.com/) and Lean Startup presentation (http://www.slideshare.net/startuplessonslearned/lean-startup-presentation-to-maples-investments-by-steve-blank-and-eric-ries-presentation).

      Share
  3. Hi Mike, very interesting post.
    It makes a lot of sense that forcing something to meet a given formula won’t be attractive to the best VC’s. Isn’t the whole point is to find something that’s not formulaic; new, different? Of course we entrepreneurs will always look for the formula to success.

    I’m curious about your response to iTrackMine – pretty critical of the big social networks that aren’t profitable – eg. Facebook. Do you think it’s a quality site? Perhaps here it’s a question of whether it delivers quality to the revenue-generating customer, which in this case is not the millions of users but the advertisers.

    Share
    1. My point was that some of these companies, no matter what they do, cannot generate enough revenue to cover pretty low costs. Clearly, Facebook does not fit the bill on that one. Nor does Twitter. Those companies have made the choice to prioritize revenue growth over profitability now for greater profitability later — a smart choice, I think.

      And I’m totally in agreement on your point that you must generate value for your customer, the advertiser. Media businesses separate consumer (consuming entity) from customer (paying entity), leaving many companies confused about who to prioritize. While it’s true that you need to serve your consumer’s well, it’s also true that you better deliver value to the person who butters your bread ;-)

      Share
  4. Great post, as usual, Mike.

    I think the reference to spam is very accurate. Spammers are the ultimate in multivariate testing and optimization. Every typo, ridiculous fact about a foreign king needing money and crazy URL layout is done with intention in a spammer’s email, to walk the fine line of getting users to click while going under the radars of anti-spam engines.

    Do they have traction? Hard to say. Though they do make a lot of money. :)

    Share
    1. Great point Nick. Spammers are the original multivariate masters and many make a ton of money. Many bank robbers also make a ton of money ;-)

      Share
  5. Mike,

    I think you are focusing too much on the use of machine learning and multivariate testing to get signups at the cost of a far stronger point in your article (the last one): that startups should first spend time building a valuable product and then focus on optimizing it using multivariate testing. When you build something people want, any traction you get is there to stay.

    No matter how much multivariate testing you do to optimize your “copy, size, color, font and placement of data fields; the format and text of buttons; the flow of pages”, in order to get millions of users to sign up, you still have to a) Get millions of users to your web page b) At least appear to offer something they think may be valuable. Millions of people won’t sign up for free TPS reports no matter how well you design your site.

    What’s more, if the number of signups was such an important metric, MySpace would be hot hot hot. Active accounts is what matters and people actively use a product only when they find it to be of value.

    And after getting all of this right, there’s the actual goal of making money to think about :)

    Share
    1. AJ, I agree with everything you said. The reason I focus on the “hacking traction” bit is that I’ve seen a ton about getting product right, but little about how people are using very sophisticated mathematics to attract users without a product.

      Share
  6. Would be interesting to have someone show a real-life case study of using multivariate to optimize traction. That would be one of the best attended conference sessions in the valley today.

    Share
    1. No doubt!

      Share
  7. Also, while it’s fairly simple to incentivize users to register, are there also cases where you are skeptical of metrics that show repeat usage? Isn’t that harder to hack?

    Also, we have also used multivariate testing to drive registrations. But in our experience this has been an optimization with improvements measured in 10%s of percent but not in multiple factors. Would appreciate an example that shows dramatic improvements, where there are 10′sMM pageviews, where in the absence of hacking it would only be order(s) of magnitude less.

    Share
    1. I don’t want to name names — I realize my argument could be strengthened by doing so, but some friendships and relationships would suffer as a result ;-) The social media space is a good place to look to for examples — including both independent websites and Facebook applications, for example.

      In terms of repeat visits, you are right that looking at repeat visits is a very good sign of engagement. And if you ask for the right data and ask the right questions you can piece together what’s really happening. In fact, that’s one of the things I do to see if there is any there, there. But smart people can often find innovative ways of initially presenting data in a favorable way — usually excluding critical views, as I am sure you can imagine.

      Another thing I like to do is to search blogs and Twitter for signs of something interesting. A Twitter Search is a great way to quickly learn a great deal about a company…

      Share
  8. [...] The dark side of marketing optimization [...]

    Share
  9. [...] around One of the big thematic issues that has been referenced numerous times by myself, Eric Ries, Mike Speiser, and others is the limitations of quantitative testing in building a business. In particular, [...]

    Share
  10. Hmm multivariate has been around for a while and lots of big ecommerce sites use it like Amazon. As well the advertising industry has been hacking us for years :) I don’t think its the MV that is to blame for the false traction as the commentor above notes. Its what these companies do with the info they acquire from users without their knowledge. For instance there are quite a few companies that use the info they get by you adding an app or signing up for their service to figure out what other sites you are on, your friends and even your email. They then use this information to bombard you with requests on other social networks and email you without your permission or at least without making it clear they will. Another example is cross app pollination which fb has tried to kill several times i.e. I give one app permission and they sneak in alerts from another app they want you to add into the stream alerts from the one you added.
    I think its this behavior along with other, sometimes even more nefarious, behaviors that not only show fake traction but really alienate consumers. It has taken a while for these techniques to backfire but from what I see there is little or no chance of this slowing down because these techniques are *incredibly* effective. The numbers are truly staggering and just like the spammers they will do it as long as it makes them money.

    Share
    1. I’m not blaming the tool, but rather the people who think the tool is a product. As I noted in the post, I think optimization techniques are a great way to make a good thing better. But that the tool is not a product in itself.

      Share
      1. Got it, I agree. Sadly I think most of the people showing traction are using far more than MV to get the numbers at least all of them I know. The tough part is several have managed to use the ill gotten traffic to build pretty substantial businesses. Time will tell if they can leave the very techiques that got them there. Great to see posts on this stuff hope you do more!

        Share

Comments have been disabled for this post