56 Comments

Summary:

“The MobileMe launch clearly demonstrates that we have more to learn about Internet services,” Steve Jobs wrote in a recent internal email. Amen to that. In fact, Apple needs a crash course in Internet services and infrastructure.

Steve Jobs, in an internal email seen by Ars Technica, makes clear that he’s upset about the botched launch of MobileMe, Apple’s new online suite of applications that has been plagued with bugs, including being flat-out unavailable to some for days at a time.

“It was a mistake to launch MobileMe at the same time as iPhone 3G, iPhone 2.0 software and the App Store,” he says. “We all had more than enough to do, and MobileMe could have been delayed without consequence.”

Amen to that. Having been a subscriber to dot-Mac for years, I was quite upset when the service failed to work at launch. They tried to hush everyone by waiving one month’s fee, but regardless, while some parts of it are up and running, many of the problems continue.

It wasn’t till Walt Mossberg and David Pogue publicly spanked the service with their respective wet bamboo stems that Apple started to understand the magnitude of the problem.

In his email, Jobs says: “The MobileMe launch clearly demonstrates that we have more to learn about Internet services.” You can say that again. The big question in the wake of the MobileMe debacle is whether or not the company even knows how to plan for heavy load.

I have picked up some tidbits from my Internet infrastructure sources, who tell me that:

  • There is no-unified IT plan vis-a-vis applications; each has their own set of servers, IT practices and release scenarios.
  • Developers do testing, load testing and infrastructure planning, all of which is implemented by someone else.
  • There’s no unified monitoring system.
  • They use Oracle on Sun servers for the databases and everything has its own SAN storage. They do not use active Oracle RAC; it is all single-instance, on one box, with a secondary failover.
  • Apparently they are putting web servers and app servers on the same machines, which causes performance problems.

One of my sources opined that Apple clearly wasn’t too savvy about all the progress made in infrastructure over the past few years. If this insinuation is indeed true, then there is no way Apple can get over its current spate of problems. It needs a crash course in infrastructure and Internet services. Apple’s problem is that it doesn’t seem to have recognized the fact that it’s in the business of network-enabled hardware.

The looks, UI and edge devices are only as good as the networking experience — whether it comes from Apple or from its partners. MobileMe could just be the canary in the coal mine as far as the Cupertino Kingdom is concerned. MobileMe isn’t that big a portion of their revenues right now, but what happens when the problems hit the iTunes store? Imagine the uproar when your 3G connections slow to a crawl because AT&T’s wireless backhaul can’t handle the traffic surge.

It might not be a problem of Apple’s making but the company will face the brunt of the backlash. Remember, most of us instinctively blame the device first, then curse the carrier.

By Om Malik

You're subscribed! If you like, you can update your settings

  1. I think you may have a few things backwards in this article. First, the problems from launching MobileMe won’t roll over to the iTunes store or the on-line Apple store. Most likely the experience from running these two highly successful and much more reliable web services will make its way to the MobileMe services. Second, you make it sound like Apple as a whole is new at this game and that the iTunes store is some small problematic but important service, instead of the largest music distributer in America. Apple has a lot of experience in handling heavy web service loads.

    Each one of those issues pointed out sounds more like trying to rush an unready product to market, which is what the e-mail Jobs sent out is about.

    Share
  2. @rene

    if that was the case then why did they not “learn” the lessons of “itunes store” in the first place. I think they have a problem. Apple store has problems but they get masked because the company use CDNs for delivery. Apps-via-CDNs are still an emerging category and you are going to see growth in coming months.

    As for the rushed delivery – I couldn’t agree more. I think Apple doesn’t have Web DNA and need to become an “internet company” quickly. It is still thinking like a box-maker. (albeit one that makes really pretty boxes.)

    Share
  3. Om, I completely agree with your observation that Apple needs to become more efficient in delivering web services. It has been pretty bad at nearly every web service its tried to launch compared to what it achieves in desktops… And I love the last statement where u mention that the device will be blamed even when its the operator.

    Share
  4. I’ve been a .mac (and now mobile me user) for almost a year. With the exception of the storage space, I am going to have a hard time staying on with the service since I can do all of the same things (and more) with Google’s free web based apps (gmail, picasa, calendar, etc.)

    I was disappointed with the redesign that looks pretty but does not differentiate the product from their competition. One of their rare bad pieces of marketing summed it up best: “Microsoft Office: for the rest of us.”

    Share
  5. I think you make good points, though I think some are a bit overstated. It appears to me that whatever Jobs is focussing his attention on at Apple gets fixed in fairly short order. And unlike some companies Apple does seem to learn from mistakes (cf aperture).

    The techniques involved in mass internet services are well-understood, even if they aren’t a core competency at Apple at this point. Apple has the money and expertise to hire the best, buy the best gear, and make its promises a reality soon, even though it shouldn’t be “soon”, it should be “now”.

    Share
  6. Rene Stein may have a good point. In the internal memo, Steve Jobs stated that Eddie Cue, who previously managed the iTunes Store, was being promoted to handle all of the company’s Internet-related services. I think Apple has done a very good job of scaling the iTunes Store to meet tremendous global demand. We can only hope that with better management, that the situation with MobileMe will turn around.

    There is still much we don’t know, and may never know. But I’m sure that at least one former manager of the MobileMe project is now seeking a new career outside of Apple. We simply don’t know if the apparent mistakes were made at the top, or by middle managers. Was the MobileMe team given autonomy in planning its network architecture? Or did the MobileMe team use a network architecture practice more common to other parts of the corporation? Om Malik’s posting does not address this.

    While the author has some insight on the causes of this mess, we don’t have the whole story. Trying to fill in the blanks with assumptions can lead to false conclusions. What we do know is that the MobileMe team failed on many fronts, and in so doing, has given Apple a black eye. But every black eye is an opportunity for a company like Apple to learn and do better in the future.

    Sadly, in technology, there are failures. Nothing in this world is perfect. The bigger issue is how a company like Apple, Microsoft, Google or others deal with them. Being paranoid, I’ll plan for the worst. And being an optimist, I’ll hope for the best.

    Share
  7. FUDtastic, Om. Super FUDtastic.

    Share
  8. @Watzat… can you explain what you mean by FUDtastic.

    Share
  9. Almost everything your Internet infrastructure sources told you is inaccurate. I worked with the guys who built the .mac (they also did the initial builds of the iTunes Music Store, btw). They definitely know how to build scalable web sites.

    Each set of Apple’s applications has their own set of servers. Well, that’s sort of true. However, there’s a very good reason for that. The development teams are also separate and code and infrastructure changes for each application need to be separate.

    Developers don’t do load-testing or QA testing at Apple. That’s usually done by QA teams which usually report to the operations teams that are responsible for the health and maintenance of the application.

    Apple does have a unified monitoring system and it generally works quite well.

    The Oracle infrastructure being described is actually similar to what iTunes uses, but not .Mac.

    The web servers and application servers ARE on separate boxes.

    Share
  10. @Brian i have much more information about their network architecture but since it hasn’t been verified beyond one person, I can’t write it up. You make a good points, but it doesn’t take away from the fact that the mobilemess was apple’s own making.

    They have had months to figure out this whole thing – announcement was months ago. So lets stop being apologists for a service we pay for.

    By the way, the MobileMe team was comprised of folks who were on contract.

    Share

Comments have been disabled for this post