11 Comments

Summary:

[qi:gigaom_icon_cloud-computing] Earlier today, I stopped by at the Social Graph Symposium at Sun Microsystems’ Menlo Park campus. The event, which attracted some of the most well-known experts on social networks and social graphs, was organized to look at the various challenges and opportunities being presented by […]

[qi:gigaom_icon_cloud-computing] Earlier today, I stopped by at the Social Graph Symposium at Sun Microsystems’ Menlo Park campus. The event, which attracted some of the most well-known experts on social networks and social graphs, was organized to look at the various challenges and opportunities being presented by the increased socialization of the web.

And there is no opportunity bigger than the one offered by the computational needs of this new social web. As we discussed during our first Structure conference (and we will continue to discuss at our upcoming Structure 09 conference on June 25th ), the social Internet has made it easy for anyone to create, publish, distribute and consume content. Such content can range from blog posts to YouTube videos to Flickr photos to simple, 140-character tweets.

But small drops of water will lead a bathtub to overflow if the pipes are clogged, and this is the challenge faced by the underpinnings of the web. “In the next 12 months there will be a zettabyte of information on the Internet,” said Dr. James Baty, distinguished engineer, VP and chief technology officer of Sun Microsystems. A zettabyte is the equivalent of 1 billion terabytes — or nearly a billion times the data stored on the various drives in my apartment.

That explains why Sun was interested in hosting the event — after all, online social interactions are key drivers of the massive explosion of data on the Internet. To help you better understand the magnitude of growth, let me share a couple of data points from a previous post about Facebook’s photo service.

  • Facebook users have uploaded more than 15 billion photos to date, making it the biggest photo-sharing site on the web.
  • For each uploaded photo, Facebook generates and stores four images of different sizes, which translates into a total of 60 billion images and 1.5 petabytes of storage.
  • Facebook adds 220 million new photos per week or roughly 25 terabytes of additional storage.

Facebook is trying to use a smart-software approach to manage this data deluge. The story is no different at, say, MySpace, Twitter or any big social web company. More and more companies are turning to Hadoop and other software written for the ultra web. Gary Orenstein in a recent post outlined the various systems that have emerged to capitalize on the data mining renaissance.

Sun wants to understand the computational needs of a web that is driven by real-time social interactions. For the longest time, the world has been OK with batch processing of data that took hours. Not any more — for the web (and the Internet) are becoming real-time propositions. To analyze the data would mean a lot of computing horsepower.

“The computational challenge of looking at the unstructured data and mining that data is immense,” Dr. Baty said. We are at a tipping point, he said, that will see compute clusters of today bulk up to levels that would put even steroid-enhanced baseball players to shame. “We are going to go from tens of thousands of (processor) cores in cluster to hundreds of thousands of (processor) cores,” he said.

You’re subscribed! If you like, you can update your settings

  1. elizabeth phillips Monday, May 18, 2009

    Seldom do I read something and want more…I want more..I wish this was a bigger piece of steak.

  2. Leoncorp999 Tuesday, May 19, 2009

    Every body wants more!!! but do u have the appetite ?
    If u can consume everything what will be left ?
    Just my 1 cent

  3. Bit=0 or 1
    Nibble=4 Bit
    Byte=8 Bit
    Kilobyte=1024 Bytes
    Megabyte=1024 Kilobytes
    Gigabyte=1024 Megabytes
    Terabyte=1024 Gigabytes
    Petabyte=1024 Terabytes
    Exabyte=1024 Petabytes
    Zettabyte=1024 Exabytes
    Yottabyte=1024 Zettabytes
    Xonabyte=1024 Yottabytes
    Wekabyte=1024 Xonabytes
    Yundabyte=1024 Wekabytes
    Udabyte=1024 Yundabytes
    Tredabyte=1024 Udabytes
    Sortabyte=1024 Tredabytes
    Rintabyte=1024 Sortabytes
    Quexabyte=1024 Rintabytes
    Peptabyte=1024 Quexabytes
    Ochabyte=1024 Peptabytes
    Nenabyte=1024 Ochabytes
    Mingabyte=1024 Nenabytes
    Lumabyte=1024 Mingabytes

    1. you betcha 4am when you wrote that!

  4. The greatest benefit of Facebook is that it has many groups on the site that you can join. So if you are interested in Chicago Cubs you can research Chicago Cubs in the groups section and you will be able to find friends on there that like the Cubs. This is just one example, I know that you can join groups of your favorite football team, television show, or whatever you want for the most part! If you can’t find a group for your interest, you can simply create one!

    James
    http://EmailCharger.com

  5. Autopsis » Blog Archive » Meeting the Lumabyte Tuesday, May 19, 2009
  6. Gigaom’s analysis of the expotential grown of social networking | Photographs and Reflections Tuesday, May 19, 2009

    [...] Here is the rest of the story: http://gigaom.com/2009/05/18/social-networking-brings-the-zettabyte-era-to-the-internet/ [...]

  7. Why Location Awareness Will Make the Web More Useful Wednesday, May 20, 2009

    [...] at least we’ve started the journey toward it. Otherwise we’d find ourselves lost in what is going to be a massive data deluge. [...]

  8. Manish Pahuja Thursday, May 21, 2009

    Perhaps the new social explosion on web will create more data mining jobs…something to cheer about for BPOs.

    1. Interesting topic, it’s amazing how the capacity has expanded so fast. It won’t be long before the Yottabyte! There is more info on this topic at http://coolintl.com/?p=402

  9. Growing Engagement Gives Rise to Mobile Social Phones – A Collection of Latest Happening in Technology Field Wednesday, April 21, 2010

    [...] While most modern phones allow for installing social networking applications or using the web to check and update one’s status, it may not be long before all phones offer native and direct ties to the big social networks. The open question then, is: Will the data networks be able to handle the demand created by all of us tweeting, sharing photos an…? [...]

Comments have been disabled for this post