Web 2.0, Please Meet Your Host, the Internet

79 Comments

I have a major problem with many of the Web 2.0 companies that I meet in my job as a venture capitalist: They lack even the most basic understanding of Internet operations.

I realize that the Web 2.0 community generally views Internet operations and network engineering as router-hugging relics of the past century desperately clutching to their cryptic, SSH-enabled command line interfaces, but I have recently been reminded by some of my friends working on Web 2.0 applications that Internet operations can actually have a major impact on this century’s application performance and operating costs.

So all you agile programmers working on Ruby-on-Rails, Python and AJAX, pay attention: If you want more people to think your application loads faster than Google and do not want to pay more to those ancient phone companies providing your connectivity, learn about your host. It’s called the Internet.

As my first case in point, I was recently contacted by a friend working at a Web 2.0 company that just launched their application. They were getting pretty good traction and adoption, adding around a thousand unique users per day, but just as the buzz was starting to build, the distributed denial-of-service (DDOS) attack arrived. The DDOS attack was deliberate, malicious and completely crushed their site. This was not an extortion type of DDOS attack (where the attacker contacts the site and extorts money in exchange for not taking their site offline), it was an extraordinarily harmful site performance attack that rendered that site virtually unusable, taking a non-Google-esque time of about three minutes to load.

No one at my friend’s company had a clue as to how to stop the DDOS attack. The basics of securing the Web 2.0 application against security issues on the host system — the Internet — were completely lacking. With the help of some other friends, ones that combat DDOS attacks on a daily basis, we were able to configure the routers and firewalls at the company to turn off inbound ICMP echo requests, block inbound high port number UDP packets and enable SYN cookies. We also contacted the upstream ISP and enabled some IP address blocking. These steps, along with a few more tricks, were enough to thwart the DDOS attack until my friend’s company could find an Internet operations consultant to come on board and configure their systems with the latest DDOS prevention software and configurations.

Unfortunately, the poor site performance was not missed by the blogosphere. The application has suffered from a stream of bad publicity; it’s also missed a major window of opportunity for user adoption, which has sloped significantly downward since the DDOS attack and shows no sign of recovering. So if the previous paragraph read like alphabet soup to everyone at your Web 2.0 company, it’s high time you start looking for a router-hugger, or soon your site will be loading as slowly as AOL over a 19.2 Kbps modem.

Another friend of mine was helping to run Internet operations for a Web 2.0 company with a sizable amount of traffic — about half a gigabit per second. They were running this traffic over a single gigabit Ethernet link to an upstream ISP run by an ancient phone company providing them connectivity to their host, the Internet. As their traffic steadily increased, they consulted the ISP and ordered a second gigabit Ethernet connection.

Traffic increased steadily and almost linearly until it reached about 800 megabits per second, at which point it peaked, refusing to rise above a gigabit. The Web 2.0 company began to worry that either their application was limited in its performance or that users were suddenly using it differently.

On a hunch, my friend called me up and asked that I take a look at their Internet operations and configurations. Without going into a wealth of detail, the problem was that while my friend’s company had two routers, each with a gigabit Ethernet link to their ISP, the BGP routing configuration was done horribly wrong and resulted in all traffic using a single gigabit Ethernet link, never both at the same time. (For those interested, both gigabit Ethernet links went to the same upstream eBGP router at the ISP, which meant that the exact same AS-Path lengths, MEDs, and local preferences were being sent to my friend’s routers for all prefixes. So BGP picked the eBGP peer with the lowest IP address for all prefixes and traffic). Fortunately, a temporary solution was relatively easy (I configured each router to only take half of the prefixes from each upstream eBGP peer) and worked with the ISP to give my friend some real routing diversity.

The traffic to my friend’s Web 2.0 company is back on a linear climb – in fact it jumped to over a gigabit as soon as I was done configuring the routers. While the company has their redundancy and connectivity worked out, they did pay their ancient phone company ISP for over four months for a second link that was essentially worthless. I will leave that negotiation up to them, but I’m fairly sure the response from the ISP will be something like, “We installed the link and provided connectivity, sorry if you could not use it properly. Please go pound sand and thank you for your business.” Only by using some cryptic command line interface was I able to enable their Internet operations to scale with their application and get the company some value for the money they were spending on connectivity.

Web 2.0 companies need to get a better understanding of the host entity that runs their business, the Internet. If not, they need to need to find someone that does, preferably someone they bring in at inception. Failing to do so will inevitably cost these companies users, performance and money.

79 Comments

Peter

@ Allan.

We created Pingsta ICE – http://www.pingsta.com/ice_intro – for this very purpose. To provide startups, SMBs, large enterprises and service providers alike {yes, even Amazon is welcome ;-) } with Network-Intelligence-as-a-Service. This way, companies will simply plug into the ICE platform and tap into Pingsta’s coalesced Internetwork expertise on-demand – like a utility on a pay-as-you-go basis.

Cheers,
Peter
Pingsta Founder & CEO

Hone

Many of these internet operations problems are caused by the two web 2.0 business models which are:

“I’ll create a site that goes viral creating massive traffic and user base so I can sell cpm’s for piddling amounts”

or

“I’ll create a site that goes viral creating massive traffic and user base which makes no money, but the traffic and user base are so impressive that maybe some sucker will buy a percentage so I can say its worth x billions”

If their business models didn’t rely on massive volumes of traffic then this wouldn’t be such an issue.

Many of these web 2.0 companies have models that mean the more traffic they have the more money they lose.

If lots of people like it, use it, but it’s free, and it relies on contributions from ‘investors’ to continue, it’s not a business, it’s a charity – like Twitter.

Geoff Lisk

Being a self-described router hugger I’m a little biased here, but the message is simple – ignore infrastructure at your own peril. I would venture that most readers of this article do not know how (or have the tools) to replace the brakes and tires on their automobiles themselves but know that they need to be maintained and replaced. So what do you do? Hire a professional on a contract basis with the tools and skills to maintain the brakes and tires on your auto. Any web 2.0 (whatever that is anyway) company who ignores the fundamental infrastructure concerns underpinning their shiny new concept are at best reckless, at worst incompetent or even downright fraudulent. Extending the automobile analogy one final time, you wouldn’t say “a mechanic is too expensive and hard to find so I’m going to take my chances” why would you do the same with your business?

Allan, thanks again for another great article!
-Geoff

Maurice

dave

seting up a couple of routers in this sort of configuration is not exactly fracking rocket science needing a Quad CCIE

sound slike there CTO was not doing his/her job

Allan Leinwand

Sorry folks about being out of pocket today. Back to the conversation:

@David Mullings – (1) I think you can attract good router-huggers to a startup just as you attract good early employees. Either pay them well or sell them on the your ideas. (2) There are lots of good managed hosting providers – ask your peers or traceroute to your competition :)

@Kevan – thanks, I’m flattered.

@Tom – In principle I agree with you, but most folks don’t know that they even need to seek out some source of the vast font of low-level networking knowledge.

@Victor – great points.

@Antoinette – great point on the macro economic environment.

@gerry – great news. Maybe you can share with others how you got this accomplished?

@Stacey – Interesting idea. Back about 5-7 years there were consultant shops that did this but I have not heard much about this business in some time. Maybe we need a Web2.0 app that dispatches router-huggers on demand? :)

@Cham – Fair enough and point well taken. That being said, I don’t think I am grossly exaggerating given some of the conversations I’ve had lately.

@Loring – thanks for the comments.

@Tom – agreed, but you need to get someone involved who understand more than just ping, pipe and power if you’re really going to make any serious amount of money.

@Alistar – thanks very much! I’m off to register a few domain names now :)

@inthemission – thanks, always great to hear from a DI alum.

@Andrew – way too geeky, but in this case the same prefix was announced out both upstream eBGP peers. You’re not boring me :)

Andrew Mulheirn

@Allan: Yeah – I see what you mean about them not losing the internet because of the default route and interconnection, but I was more thinking of the Internet losing half of *them*.

More details offline. I’m probably boring everyone with my router-hugging antics. ;)

inthemission

The article was pretty good, the comments are hysterical. Anyone who thinks that we’ll just put something on Rackspace, or we’ll just hire some consultant to tell us what’s wrong and everything will be alright is really fooling themselves.

How do you know what your consultant is telling you isn’t a complete bunch of garbage? “Gee, I tried starting my car and it sounds like the engine is trying to start but then it kinda winds down and stops”. “Well, you obviously need new brakes sir!” “Oh, ok, here’s my credit card”.

Relying on someone else’s expertise without knowing anything about the subject matter is a good method for parting you with your money. Most Web 2.0 startups don’t have much of the green stuff to waste.

Good sysadmins or network engineers are essential for keeping your site from getting hacked, overwhelmed, horribly underutilized etc.

You think your sales rep at the hosting company is going to say “Why are you running CUPS on your hosted server? Maybe you should tune your box before you pay us more money for another machine”!??!

Because someone runs a company that does managed hosting doesn’t mean everyone that works there is smart or good at what they do either. I called IBM once to say the ISP I was working for was having trouble with the SMTP connection to a certain bank of their MX hosts. The response of the guy on the other end? “Have you checked your POP settings sir?”

It’s up to you to be an educated consumer of the services you are buying. If you don’t have time to get educated, then hire someone who is educated (sysadmin/network engineer/what have you). Running an Ubuntu box at home is not the same as being a professional sysadmin, trust me. Plugging in your DSL modem or even FIOS connection, for Pete’s sake, doesn’t make you a network engineer.

Amazon S3 went down for us about 3 weeks ago, by the way. So much for that theory.

Thanks for the great post Allan. Kinda sorry I got to DI a few months after you left.

Alistair Croll

Allan,

This may be my favourite post of the year.

I would argue that just as computers are giving way to computing, routers have given way to routing. I have no desire to hug a router, or even a CLI for that matter. But I have a tremendous respect for people who understand routing.

The issue here is abstraction. You can’t function in AJAX, within HTML, within HTTP, served by Ruby, run on EC2, sitting atop a hypervisor, and so on, without losing touch with the fundamentals. This Internet thing is still about packets getting where they’re going, regardless of how shiny the buttons in those packets are.

You also touch on a related point — scalability used to mean computing and bandwidth. Now it means social backlash, comment spam, community management, and so on. We can’t just register .com, .net, and .org any more; now we need to make sure the Twitter, Drop.io, Myspace page, and Facebook names aren’t taken too.

Great piece.

Tom Sparks

I’ve seen quite a few of these situations, and ultimately a lot of it has to do with developers ignoring the inevitability that their site will expand beyond one server and a couple users. The Amazon services don’t save you from this, and I think it would be foolish to rely on that as ones scaling plan. Even some managed hosting providers fail at providing loadbalancing, so if you’re starting to make real money it pays to make a minor capital investment into getting colo and some folks who know how to do things.

David

Emil — wrong, to be honest. Infrastructure concerns should be thought about from the offset of a project. Not scaling properly can kill a project (or their budget, later on) so it should be taken into account from the ground up.

The other half of it.. bad code, could kill any worthwhile project as well. Some survive nonetheless but taking them into consideration upfront will save tonnes of hassle! :)

On another note, there’s a widerange of inexpensive hosting providers (some scale well, too) on http://www.hostjury.com :)

John

I cannot believe this made it on Digg.
What a waste of time by people who think they know what is going on.
If you are having a hard time logging into Facebook try something else.

Loring Wirbel

Stacey’s post is a great idea, but somewhat misses the point. The vast majority of humans taking advantage of infrastructures (not just communications, but power-grid, etc.) dwell blithely and happily at Layer 7, thinking that problems arising at anything under the application can be solved through liberal applications of pixie dust and magical incantations. Not to sound too much like a survivalist, but it behooves all of us to understand the basic operations of all underlying infrastructures, preparing for the day when things fall apart, and we have to go back to configuring our own routers and manually setting the f-stops for our own non-digital cameras. Maybe even slide-rule education would be useful. Pixie dust just doesn’t cut it.

Cham

It’s very hard to take this article seriously when it uses gross exaggeration to make a point. To say that Web2.0 companies “lack even the most basic understanding” is an offensive exaggeration. This statement seems particularly dumb since you follow it up with a description of how a team specialists were required to resolve the problem.
It would be enough to say “routers are too often over looked…” to make a point. No need to slap us all in the face to get our attention.

Stacey Higginbotham

Allan, perhaps in this column there’s the germ of business idea. Much like their are several virtual CFO companies, perhaps a network engineer SWAT team business would do well for startups. Like everything 2.0, it would be “on-demand.”

gerry

We have an elegant network based defense against DDos and other hostile attacks.

Antoinette

It seems like a lot of these companies are being driven by overly high expectations of striking it rich without a solid foundation in the evolution of the Internet over the past 15-20 years. It’s like 2000-2001 didn’t even happen. I read a lot about Web 2.0 because I’m interested in how technology changes affect society and most of stories are not about technology at all but had to make a profit off of Internet development.

I don’t have a problem with people trying to make a living or even getting wealthy off of innovation as the profit motivation has driven a lot of internet development. But I think a lot of these folks have completely unrealistic expectations about striking it rich and I think another crash is coming in about 9 months or so unless new enterprises have a firmer footing and grounding in the cycles of internet growth and slowdowns. Their ambitions seem disconnected from the goals and abilities of developers and programmers who actually create the programs they are trying to make money off of. JMHO.

Victor Blake

Finding router huggers — it’s easy. You PAY for them, just like you have to pay for everything else in business. It’s not free. Turns out that the strongest network engineers are also past-developers. Not really amazing since routing started it’s life as an application — written by telecom engineers…

Seriously — your board and advisors should help you find one key Internet architecture, security, and engineering person to advise your company. You will need to do some combination of compensation including one or more of stock/equity, consulting, or both.

And I agree with some of the other posters that to keep costs down and your focus tight, you do not necessarily need to hire full time staff (depending on what you want to do). I will say this. The right person can help with foundation architecture work as well. Building network constructions into the software architecture could save you a lot in the long run.

-Victor

Victor Blake

Major sites that I’ve worked on can log easily in the millions of attacks per IP address PER DAY. Depending largely on the number of source attacking hosts (or their shadows from say a SMURF attack) it can add up fast. My point is that the incumbents have to sustain themselves against these attacks 24×7. So why should it be any different for the poor lowly 2.0 startups — they need to secure the applications as much as the incumbents do. I agree completely that every company that relies on the Internet as the PLATFORM for their product (not to sell the product — but to RUN on) should have in-house expertise or closely held advisors that can provide guidance especially on matters of security.

Comments are closed.