Blog Post

Like modern web apps, start-ups are distributed systems

Another highly distributed system.

I’ve been thinking about distributed systemsa lot lately. While the last decent code I wrote was in Logo, it suddenly dawned on me that I deal with a particular type of distributed system every day: start-ups. Let’s examine a few of the similarities.

Modern web-scale applications like Google,(s goog) Twitter, Netflix,(s nflx) LinkedIn,(s lknd) etc. are implemented as distributed systems as opposed to single monolithic codebases. This means that they are composed of tens or hundreds of services that communicate asynchronously with each other, ultimately delivering a response to the end-user. is composed of more than 200 services, according to a talk by Jeff Dean in 2010. Some of these services are external facing, spell checker, instant search, etc., while others are only for internal consumption and not visible to users. Not to get too meta, but most of these individual services are in fact distributed systems themselves.

Philosophical and physical distribution rules at startups

Develop at your own speed.

One of the reasons the service-oriented approach has become dominant is that each service can be developed, managed, and scaled independently. It also allows teams to iterate and update individual services at their own pace.

Here, there is a direct parallel to the functional structure of start-ups and large companies. Engineering, marketing, sales, and finance are their own disciplines, have their own teams, and move with their own cadence. Actual organizational power, decision-making and relationships are complex and seldom map to the org chart. Similarly, what a web-scale application really looks like is a messy set of interconnections, seldom mapping to the 3-tier architecture diagram.

Start-ups are also becoming more physically distributed. Engineering in Israel, sales in the U.S., or the rock star developer who lives in Boulder/Tahoe/Iowa because he/she can. This is so common as to barely merit mention. Overcoming physically distributed teams is easier than it has ever has been assuming a commitment to communications.

Skype, Chatter, Jive, IRC, IM, etc. can all enable distributed companies to scale. Over communicate but also know the limits over technology and be realistic about what needs to be done face-to-face. A great read on this topic from Basho who are experts in distributed systems and whose company is a distributed system can be found here.

Everything’s federated: From social buttons to SaaS

Most web pages now contain content that comes from a variety of third-party entities. Examples of this behavior include content delivery networks, performance monitoring tools, and ad networks. Start-ups are very similar here, as well.

There are a variety of third party SaaS services required to run their businesses, ranging from Github to (s crm) to Marketo. Failures in any of these internal or external business services can impact the business in a similar way that a third-party party ad network can slow down your page load times.

Scaling: It’s hard for sys admins and for startups

Large-scale distributed system like Twitter, Google or Facebook, (s fb) have evolved their software architectures significantly to deal with the challenges of scaling. The path from hosting a site in your bedroom, to Amazon Web Services, to a col-location space, to a data center you own requires bringing in new skills that were previously not relevant to help along the way.

Mirroring that, the organizational structure of a start-up will change significantly as it grows. From the early days of product development, through learning how to sell and adding a customer-facing team to replicate early wins, each stage will require organizational changes and evolution. In both cases, changes need to be made on the fly and require careful planning and execution. Even then this can be painful; the often-used “change the engine while the airplane is in flight” analogy.

Herding cats might be more predictable

Both start-ups and distributed systems can have unique, unpredictable, even Byzantine failure modes. In each scenario because of the complexity of interactions, there are bound to be situations that can’t be tested or even anticipated.

This is further complicated by a complex set of dependencies on actors you don’t directly control such as the third-party services mentioned above. Even though services are developed independently, connecting them all together often results in failures that have a way of cascading. This is easy to see in the service interruption of a highly complex distributes system like Amazon’s EBS service a few years back.

In start-ups, the “failures” or “outages” can be things like a product release slipping, a disgruntled employee negatively impacting culture, sales ramping slower than expected, or fund raising taking longer than expected. Many of these can be recovered from, some cannot. I’ll cover start-up failure modes, disaster porn, and how to avoid them in a subsequent article.

Scaling distributed systems and organizations is hard. My dime store organizational behavior observations aside, I’d love to hear about other similarities and difference that come to mind.

Alex Benik is a principal at Battery Ventures who invests in enterprise and web infrastructure start-ups. You can find him on Twitter at @abenik

Ant image courtesy of Flickr user me’nthedogs.

5 Responses to “Like modern web apps, start-ups are distributed systems”

  1. tleach

    GREAT POST!! It’s very interesting way to think about startups and the technology we use today. It has caused me to recalibrate my thinking about my startup.

  2. Donald Rippert


    Great post with two very important points.

    First, the relationship between distributed systems and the ability to develop applications quickly. Developing applications out of services has been something of a holy grail over the years. People knew that service-based applications could be developed and evolved more quickly than monolithic applications. However, everybody worried about managing the latency of the application if those services were built on a monolithic infrastructure. Distributed architectures not only allow applications to grow to web scale they also allow sophisticated applications (like Google.Com) to be built and enhanced quickly and cost-effectively by making latency management less tied to application development.

    Your second point about distributed companies is also well taken. You reference Basho and link to a blog by Mark Phillips. Thanks for the mention. We here at Basho are committed to distributed systems from a technical and organizational perspective. On the organizational side, we simply would not have been able to build the same high quality team in any single location as we have been able to build with a distributed approach. There are certainly challenges but attitude, culture and tools help us overcome those challenges. For example, our use of Yammer shows how modern tools make our distributed organization work.

  3. Dave Sandrowitz

    I think you are spot on in building this analogy, at least when it comes to the complexity and expansive set of risks that each face. However, I think distributed systems have easier pathways to address some of this given the (hopefully) inherent scalability built into them. Startups have some scalability they can take advantage of at a technical level, but very little that they can employ when it comes to human capital. Three co-founders and an intern just don’t scale well, particularly in the middle of a product release cycle or a fundraising effort. I know you touch upon all of this a bit, but my take is simply that scale is really where this breaks down, at least if you consider early stage startups that are pre-revenue and pre-investment (maybe even a it post-seed round).