Blog Post

MySpace Down Again

Update #2: MySpace is back online and whatever the issues were seem to have been fixed. MySpace spokesperson blamed the outage on power outages caused by unseasonably hot summer in Southern California.
A power outage seems to have hit the data center that hosts, the largest social network in the world. This is the second day in a row we have heard similar reports, though today we have managed to grab a screen shot. Ironically, the still shows an ad – at least the FIM has its priorities right.

But seriously, it is hard to believe that a service this large could just have one data center. Have they not heard of redundancy? I am pretty sure there is more to the story. One can only imagine how millions of MySpace users feel right now. Mashable has more information on this developing story.

Update: Jake says that MySpace is in the Equinix data center.

Mark Fletcher explains the technical issues involving MySpace and other social networks, and why MySpace could be in one datacenter.

128 Responses to “MySpace Down Again”

  1. all the news channels are reporting that the message posted on the temp page “hopefully back online within the hour. it’s 6:40pm PST now” was posted sunday at that time… that is so not true… THAT was the same message posted saturday evening during the first round of problems, which lasted until around 3am PST… the site was mostly functional throughout the morning sunday, and it totally went down around noon… the temp page was put up shortly after that, and the message from saturday was added around 9pm PST…

    hour my ass…

    btw, the temp page is being served out of the AZ data center…

  2. Dimitar,

    Thanks for the link, quite informative.

    The money quote in it:

    Equinix Exchange is an Ethernet-based peering service that connects MySpace to all of the top-tier Internet backbones through a single hub — a ‘fabric’ of server switches connecting all Equinix networks

    In spite of all the fancy hardware, that introduces a single point of failure, which is a damn stupid thing to do.

    It is always a good idea to partition large volume infrastructures in terms of DNS, middleware code and databases, aka, don’t put all your eggs in one basket, no matter how huge or well-designed the basket is, because all baskets fail at some point or the other.

    Strangely, the domain resolves to two different sets of IPs in CA and AZ. So, they are distributed in a manner of speaking.

  3. ~~*CHELLE*~~

    omg.. myspace :'( i need it… lol.. joke… but seriously myspace took me ages to make and set up.. and all the people on there are my REAL friends from school and stuff but i do agree with talking changa.. we mite all have to varify ourselves and possibly pay! wat a disaster!

  4. why does this have to crash, i dont understand how they loose the data, its online, so surely its quite hard to actually loose that data, and i hope that that doesn’t mean that we will all use our myspace profiles e.t.c…

    Oh my god can you imagine if that was to happen, myspace starting all pver again, oh dear that WOULD be hell for sure.

    Amy x x x

  5. can u imagine if myspace never came back? all those surveys, blogs and comments will be LOST FOREVER!! oh no! for the mean time, everyone is just keeping themselves preoccupied by making REAL friends, talking on the phone or looking at anime porn (that’s what happened w/my friend and he told me to check out heinka and i freaked out bc i didnt know what it was) ANYWAY! i think there is more to this story; i bet myspace will start charging once it comes back up. It’ll also have us enter our ID # to prove that we’re not a teenager using myspace to whore ourselves.

  6. How MySpace Works

    “# Database: MySpace has a 20-terabyte (TB), scalable cluster of Isilon IQ 1920i servers. (A terabyte is 1 trillion bytes.) The cluster stores users’ uploaded media files (images, videos, songs), among other things. It runs on 3.2-GHz Intel Xeon processors and has 10 server nodes storing 1.92 TB each. The nodes communicate on the InfiniBand architecture, which establishes point-to-point, serial connections between each server node. Isilon reports that data-transfer speeds are in the neighborhood of 3 GBps.

    Back-end: MySpace’s back-end is an InfiniBand server switch. A server switch facilitates communication between multiple severs on multiple platforms — in this case, between all of the servers in the MySpace infrastructure and between the MySpace servers and the external servers that deliver content to the end user. The Equinix Exchange central server switch in Los Angeles hosts MySpace’s content delivery. Equinix Exchange is an Ethernet-based peering service that connects MySpace to all of the top-tier Internet backbones through a single hub — a ‘fabric’ of server switches connecting all Equinix networks. Content travels from MySpace through any of the Equinix-connected networks based on the most efficient route to the user and the available bandwidth on a network at any given time.”

  7. Maybe I’m getting old, but this seems to happen every summer. I’ve seen a lot more people seriously talking about disaster recovery in the last few years, and locating the site that pays all your bills in a data center in California seems like a bad idea. California has a power shortage and no visible plans to solve the problem–even if they did it would take years to catch up to demand. What happens to your business when a huge earthquake hits the bay area? If it were my business on the line I’d be somewhere like Chicago.

  8. wafermouse: firefox users with the adblock extension can disable the ads… and with greasmonkey scripts, you can make the interface to suit you… and you can disable the autoplay of the music in Home~Account Settings~Music Settings…

    brent: firefox has an extension called “reloadevery” which allows you to set your browser to automatically reload the page at a specified interval (mine is set at 5 seconds)…

  9. WaferMouse

    Debra: The difference there is that WordPress was actually planned out. MySpace was a purely accidental success. God only knows what billions of teenagers and every band that can string 3 chords together sees in the horrible interface, flash ads, flash music players that play automatically, etc. etc.

  10. rely heavily upon their (constantly updating) databases and by their nature are non-trivial to scale and/or replicate across datacenters.

    Doesn’t Amazon maintain consistancy across databases hosted across data centres? If they can, why can’t MySpace?

  11. this is interesting to me, given the number of users myspace, this would be the equivilant of something like Yahoo or Google going down for a day, which is unheard of, but people are so used to the bugs in myspace that it seems almost normal that its down.

    I must admit though, it is very complex and must be difficult to run.

  12. Jake,

    You can do remote replication, just get a couple of dedicated links and you’d be ready to go., with probably 1/400 resources that Myspace has already does this.

    That said, I am just guessing here without having any knowledge of the inner workings. So I would cut them some slack.

  13. Om, I might be being naive here, but that looks like a Flash game, not an ad. So I bet those millions of MySpace users are saying “damn, I got eaten by a ghost again.”

    Also, if you view source, the comments (from past outages) are sort of funny. :)

  14. Social networking sites rely heavily upon their (constantly updating) databases and by their nature are non-trivial to scale and/or replicate across datacenters. I wouldn’t be at all surprised if MySpace was only in one datacenter. Many (most?) Internet services are not replicated across datacenters.