46 Comments

Summary:

The reasons for the recent screwups by Apple, Sony and Amazon were different, but their reaction was remarkably similar: a conspicuous lack of timely response. Like many others, these tech giants don’t seem to have realized that crisis response has to become real-time now too.

We’ve seen a trifecta of failures and/or screwups over the past couple of weeks, from three of the world’s technology giants: Apple was shown to be keeping a log of the location of millions of iPhone users, Sony’s PlayStation platform was hacked and millions of customer accounts were compromised, and Amazon’s (a amzn) cloud-hosting service EC2 went down for hundreds of companies. The reasons for these screwups were different, but the reaction from the companies involved was remarkably similar: a conspicuous lack of timely response. Like many others, these tech giants don’t seem to have realized that crisis response is real-time now too.

Apple, of course, is notoriously close-mouthed about this kind of thing, as was seen in the “Antennagate” affair last year, when Apple stonewalled on the issue of its new iPhone 4 antennas and flaws in the unit’s design. And the initial response from Apple co-founder and CEO Steve Jobs in that case was also classic: he reportedly said “Don’t hold it that way.” In this case, there was virtually no response from the company whatsoever to the location-data issue for days, until — again, in classic Jobs style — a statement was made to a single news outlet effectively saying the issue was overblown and/or the result of a few bugs, and that everyone should just calm down (my colleague Bobbie Johnson disagrees). Oh, and a response took so long because it’s really complicated stuff.

Given its past behavior, it’s possible that Apple is beyond help in this area. The company’s approach seems to be that people will unfailingly line up to buy its products regardless of how it handles such PR gaffes, so it may be a lost cause. But Amazon and Sony arguably have a lot more to lose.

Sony in particular — a former technology leader — has not been doing well on a number of fronts for years now, as Apple has taken over virtually every market segment that the technology company used to own. Not only that, but the company is already infamous in computer security circles for its last major fiasco in 2005, the “Sony rootkit” affair, in which users had a virus-like software program installed on their computers without their knowledge if they played a CD. So you might think that the company would try hard to get out in front of the most recent issue — which venture investor and technology analyst Paul Kedrosky described as “among the worst such debacles in modern financial/technical history” — as quickly as possible. Oh, but it’s really complicated too.

Amazon CEO Jeff Bezos

Amazon is not nearly as desperate as Sony, but the company has still pinned a lot of its hopes for the future on the success of its cloud-hosting and cloud-based services business, and seeing hundreds of major companies and websites fail — and lose critical data — is a huge issue. And yet, while Amazon eventually did release something that was much closer to an actual apology than anything Sony or Apple came out with, the company still avoided discussing the issue for what seemed like an eternity in Internet time. One Internet analyst said that Amazon’s “anemic public response” was a major flaw, and that arguably wasn’t the only one.

This isn’t an issue just for Apple, Sony and Amazon — it’s something that companies of all kinds are still struggling to deal with. The reality is that social media such as Twitter and Facebook have increased the ability of customers and users to speak out about such issues, and decreased the amount of time that companies have to deal with them. And that means the old approach of taking days to hold “war room” meetings and come up with elaborate PR plans just doesn’t work any more.

Those things still have to be done, but they have to be done a lot faster, and while they are being done someone has to respond, even just to say “We are sorry, we are looking into the problem” (and if you don’t know when to respond, try this flowchart the U.S. Army came up with for responding to blogs). You don’t just get to reap the benefits of real-time when you are a technology company: you also get to see the other side of that double-edged sword when you screw up. And it cuts just as deep.

Thumbnail photo courtesy of Flickr user Hans Gerwitz

You’re subscribed! If you like, you can update your settings

  1. Oh come on Mathew, a company can’t respond to every Blog, Dick, and Mary. Neither Apple nor Amazon will suffer from taking their time, never will. Sony, maybe because it was credit card data, but not because they took their time.
    I can understand why the media wants instant, real-time crises management, it feeds their system. Doesn’t buy a company much unless it’s a real crisis like tainted Tylenol.

    1. Totally disagree. Maybe it will hurt Amazon over the longer term, who knows — and Apple, as I mentioned, seems to be immune to any damage from this kind of thing. That doesn’t make it right. And they don’t have to respond to every blog or tweet — but no response when something of that magnitude is happening is just dumb.

      1. I actually agree with Matthew.

        I think what he is trying to say here is not that companies need to come out and explain exactly what the problem is within 10 minutes of the problem occurring, but instead to atleast ADMIT there IS a problem, and that it’s being worked on.

        Instead, companies such as Apple say absolutely nothing for weeks, sometimes months, making it look like they couldn’t give a damn about their customers or the problems at hand.

        At least come out and say “Hey, we’re experiencing some problems which we are working on. We will update you when we have more”.

        Something as simple as that will go a very long way in the eyes of a consumer, because it shows the company DOES care, and is doing all it can to find a fix ASAP.

        That was my take on this story anyways.

      2. Magnitude only describes the piling on by blogs/media, it doesn’t describe anything about the Apple non-issue. It might describe the Amazon situation somewhat and certainly the Sony mess.

        I actually fine the fact that they don’t knee-jerk every time some issue comes up. We get enough of that with the link-bait headlines.

      3. Sony lost my credit card data, and took a week to tell me. Sure it takes time to verify.. But..
        Apple wrote software that stores personal data to a personal device . If I want to lock the device I add a pin, if I feel as though my machine and user are not secure enough , I have the ability to encrypt backup of said personal information. Frankly, I am pissed that a great resource will be diminished , because some people say “it’s complicated stuff” with a touch of sarcasm and want me to explain How vunerable EVERY machine is when it’s in someones hands.

  2. coolrepublica Saturday, April 30, 2011

    Apple and Sony did wait a while to say anything, but Amazon was different and they did everything the way it was supposed to be done in my opinion.

    When the Amazon servers went crazy in N. Virginia they gave updates as to what was happening to the best of their abilities. No one can expect these engineers to be tweeting 140 characters every minutes about the problem since they are needed to put out the fire. When a fireman is trying to put out a fire in my house I could care less about what went wrong, I want him to put out the fire and tell me later. Amazon did just that. All hands on deck. They put out the fire and then they gave you a small novel detailing what went wrong and an apology. Perfect!

    Apple on the hand gave you nothing, and when it did come out with something it was in the line of “hey everybody else does it.”

    Crisis response does not need to be real time. It’s ludicrous. That’s how mistake are made. Companies should not jump into giving information that may not be accurate just to keep attention deficit reporters and bloggers happy. they need to keep people up to date but it doesn’t have to be or either should it be real time. Nobody can accurately report problems with good perspective in real time.

    1. And you know what? It won’t make any difference. You can feign outrage if you must cool republica, as you do elsewhere, but in truth it ain’t going to stop Apple’s momentum one tiny little bit.

    2. I like the “putting out the fire first” analogy and agree with what you say. Besides, if they were giving info real-time, there’s all the possbility that the media would pounce on them that instead of acting on the problem, they were wasting time tweeting!

  3. Yeah, more 20-20 hindsight armchair quarterbacking from a journalist.

    I’ve been watching Sony most closely and here’s the timeline I see:

    First, the systems were compromised. Sony discovers it and immediately shuts everything down to prevent further damage. They announce that they’ve had an intrusion virtually immediately and get to work assessing the damage.

    As soon as they find out that personal data was taken, they report it and begin the onerous process of sending out 77,000,0000 emails to members of its service.

    And since they there have been regular follow-ups. Tomorrow Kaz Hirai is holding a press conference to talk about the situation.

    Now looking back it’s easy to say “Wow, they waited too long to tell us data was stolen!” and that’s an easy claim to make now that we know it was stolen, isn’t it? But what if they’d said “Your data was stolen” right away, and it wasn’t. Suddenly a rash of people are canceling credit cards…then Sony says “Oh, great news, your data wasn’t stolen after all.” and you journalists would be trying to crucify them for being alarmists and causing people unnecessary headaches.

    It’s unfortunate it took them as long as it took them to identify the scope of the problem; they’re certainly not perfect and it would’ve been nice if they’d been able to point that out immediately…

    But in the long run, what real difference would it have made if you’d know your personal data was lost 4 days earlier than you learned it? The info has been stolen, there’s nothing we can do to ‘get it back.’

  4. Crap whining will not improve security or impress technologists designing software. Deliberately casting tech designers in the NSA mold may get more links from conspiracy cretins. About all this adds.

  5. I would not put Apple’s current PR issue in the same boat as Amazon or Sony. Amazon and Sony’s issue caused major issues, data lose and release of personal information. Outside the Tech world no one really cared about the Apple’s location tracking issue or even Antenna gate. You definitely can’t say the same with Amazon and Sony’s problems.

    I think Amazon is going to hurt the most, they LOST data on a service that succeeds on the reputation of reliability.

    1. I don’t think that’s true about Apple — I have had a number of people ask me about the location tracking, and none of them are tech types. Thanks for the comment though.

      1. Of course people would care if Apple was tracking them.

        Your ignoring the fact that that claim was made up.

        If they respond quickly to every made up claim who will it help?

      2. And yet well after the fact you STILL get it wrong about Apple:

        “Apple was shown to be keeping a log of the location of millions of iPhone users”

        No, no it wasn’t. There was a log of cell towers and WiFi hot spots, some dozens of miles away from the actual users. Apple doesn’t have a problem; “real time” reporting caused a so-called crisis where there wasn’t one.

  6. This article would be a lot more meaningful if you actually explain *why* the companies concerned should respond in real-time every time there is a media panic.

    In Apple’s recent case, the allegations were false, and the explanation was already in the public domain, and the panic was the result of journalists failing to check facts.

    The response they gave was measured and forthright, and given that they undertook to change their software, must surely have required some actual consideration.

    News is now ‘real-time’ partly because journalists don’t take the time to check facts anymore. You seem to be claiming that we’d be better off if companies didn’t bother either.

    1. “the panic was the result of journalists failing to check facts”
      B I N G O

    2. With all due respect, “in the public domain” doesn’t mean anyone knew about it — and their response was not measured and forthright, it was late and mealy-mouthed. Thanks for the comment.

      1. Isn’t it the responsibility of journalists to discover facts?

        The reason why people didn’t know about the information is that instead of looking for facts (that incidentally didn’t take much work to find), the media instead spent their time repeating false claims without understanding that they weren’t supported by the supposed research.

        Apple did the right thing by not responding to the people crying wolf until they were ready.

        Imagine what would happen if they rewarded every false accusation with a hasty response?

        If you want to critize someone’s handling of this affair, it should be the journalists.

        Quite frankly the only reason the media is griping about Apple’s response time is that they are bitter that they didn’t take the bait.

      2. Nobody knew because the researchers and media didn’t do their due diligence.

  7. Of course all these “statements” take a while to come out, spin isn’t easy… need to consult psychologists, and marketing, and the bean counters and… Of course, the truth is always instant, but “that’s ridiculous, our users can’t handle truth, they want a pleasant lie”. Did they learn it from Government, or did government learn it from them? Oh, I forgot they’re all the same now

  8. Gerald Buckley Saturday, April 30, 2011

    So long as Wall Street doesn’t pummel them for answering a day slower than you might like, they’ll answer when they’re good and ready.

    Point in case – Jobs’ health prospects and Apple’s CEO succession plan. We don’t know because shareholder value doesn’t seem to care (much… or enough… Take your pick).

    I think in all the cases you cited… It was clearly stated, the companies were trying to ascertain what the problem was and get it resolved. To expect a minute by minute tweet of… Well, we just check all the memcaches and blah, blah is (frankly) more signal than we need. Amazon did a SPLENDID job of detailing the problem. They OWNED it from the opening.

    How has Apple been punished for being deliberate in their response to crises? Seems to me ALL critical measures that matter seem a-ok with how they’ve handled antennagate, white-iPhone-gate, Jobs’-health-gate, successor-gate, etc. Maybe I’m not seeing the other sharp edge you are though…

    My whole point boils down to – a crisis warrants deliberate attention and more than 140 characters. Real time crisis disclosures mean someone is wasting valuable time NOT solving the problem.

    1. To be honest, I don’t think “they’ll tell us when they’re good and ready” is a great approach, and certainly with Jobs’ health issues (as I have written before) they waited far too long and even then didn’t provide the full information necessary until they were effectively forced to. That’s hardly a gold standard for shareholder communication. Thanks for the comment.

  9. Who is simple-minded enough to be placated by “Sorry, we’re working on it”? Without such a statement, do you think these companies are doing nothing?

    I’m perfectly happy to give companies time to respond if the response is appropriate. If the response doesn’t solve or abate the problem, then I have a problem. Not when it takes 2 days or 30.

  10. The real story here is that the tech media crossed a new line with the Apple controversy.

    Antennagate was exaggerated, but it was based on truth – calls really are dropped if you hold the iPhone in a certain way. It doesn’t happen often enough in practice to matter, but that takes data gathering and statistics to find out.

    LocationGate wasn’t just an exaggeration. Many of the reported statements and even headlines were completely false. Just checking the original research blog was enough to discover this, and a few questions to any developer who’d watched Apple’s videos on the subject would have revealed this.

    Don’t you think manufacturing a fake scandal is more harmful than Apple taking a few days to respond to a non-issue?

    1. “manufacturing a fake scandal”
      Oh, they’d never do that – Oh wait, they do need a steady flow of link-bait and page views.
      Don’t get me wrong Mathew, I love Giga stuff and you guys are less prone to this.

      1. I don’t think it was manufactured at all, frankly — I think it was a fairly complicated technical issue that was not communicated well by Apple. Responding more quickly would have kept some of the more hysterical coverage to a minimum, which is part of my point.

      2. In Apple’s case it wasn’t the story that was manufactured, it was the portrayal as “scandal”. The reality is it is somewhat a none issue and if the researchers/media had looked, it would have been even less.

      3. “would have kept some of the more hysterical coverage to a minimum”
        I’m betting it’s more like the birther thing – Will continue on even though a birth certificate has been produced.

  11. Dropbox, a company that gained popularity through good PR and Social Media Marketing, has recently tried to respond a technical crisis using PR spin. When a company resorts to using PR spin to solve technical issues you can be sure the reactor core is melting it’s way towards the center of the earth.

  12. Apparently despite not only logging your location, but transmitting it back to Google Servers, Google gets a bye. Stay classy, Gigaom.

  13. Gerald Buckley Saturday, April 30, 2011

    Given your need for speed Mathew… What modern crises would you give an A+? Just curious if you grade on an impossible curve perhaps? Or, if something of material importance has lived up to this lofty need for rapid fire, real-time disclosure AND accuracy. Exhibit A please?

    1. It’s hard to come up with examples of any large company that has done it at A+ levels, because this kind of thing is so new, but I think Toyota did not a bad job when the sticky gas pedal issue first surfaced.

      1. Wonder if the first half dozen or so people whose family members experienced sticky accelerators would agree that Toyota did an A+ job of expediently disclosing THEY had a problem of their own making on their hands?

  14. Thank you thank you. Glad someone is holding them accountable. Add Epsilon to the list, they knew about the breach earlier in the week and held it til Friday afternoon.

    To the folks that disagree, I will say this: when it comes to security issues, brands that don’t respond in a timely manner lose my trust as a customer, and the trust of reporters who cover them. They don’t have to respond to every reporter or blogger personally (except that they should), but they do need to let customers know what has happened, what they are doing to fix it/help them recover from any damages, and what steps they are going to take to make sure it doesn’t happen again. And most importantly, that they are SORRY.

    It rarely takes more than 24-hours to know know most of the answers. And if it does, companies can be transparent and report what they know along the way. It’s not rocket science.

    1. Thanks for the comment, Allison — well said.

      1. Who’s holding the press accountable when they fail to fact-check?

    2. Yes, if it’s a security issue – Sony & Epsilon were. Amazon wasn’t, nor was Apple in reality.

  15. In the case of Sony there is the risk of credit card fraud and very personal data in the hands of hackers. And Amazon’s faux pas affected companies and their income. Clearly those are more severe than the contrived locationgate scandal that you and the media at large are using to gain traffic and eyeballs.

    I’ve noticed that you spent a lot of time in the comments and the post bashing Apple. You felt so inclined to go after them that you put their name first in the list even though the issue with Apple was not as severe as Sony or Amazon by any measure. You then perpetuated the myth about Apple storing location data for users and this is simply not the case.

    The data that was stored was cell tower and wifi locations in the vicinity of the user and not the location of the user in question. You may argue that it is the same but it is not by any means because the radius from the user location (which wasn’t stored) to the nearby cell towers and wifi is relatively large and gives the user many square miles of breathing room.

    I looked at my own data and found huge discrepancies between where I knew I was and the locations that were found in my own phone. While I was in San Diego my phone had data points that were 5 to 10 miles from where I actually was and I immediately knew they were not reporting my true location. I also found errant records in states I wasn’t even in or near like Nevada. I value my privacy but I also enjoy using location based apps that respond in seconds rather than minutes.

    ChaosDNE raised a good point. I want to add beyond that if someone have my laptop or phone I have bigger concerns because they have access to my email, my contact lists, my work. The places I was near are a much smaller problem at that point. In addition to a PIN and encryption Apple also allows us to remotely wipe a phone if it is lost or stolen. Apple did provide mechanisms for security and if the user chooses not to use them then shame on them.

    It’s clear that you have an issue with Apple but for the sake of professionalism you should consider putting the apples with the apples and the oranges with the oranges if you catch my drift.

  16. Vanessa Williams Sunday, May 1, 2011

    I think both of the main points of view here: Matthew’s that companies respond too slowly, and some commenters’ that the media is part of the problem, are valid. To Matthew’s point, once your company’s product or service is getting bashed in the (real-time) media, it’s past time to mobilize and get a response out there. If customers are getting real-time “WTF?”, you need to provide real-time updates, even if there’s little substance to them. It’s the appearance of paying attention that matters. And you don’t need to distract engineers for the gory details, a simple PR department “we’re looking into it, check back for updates” is sufficient. Every company I’ve worked for, this has been SOP.

    On the other hand, it is the media that determines which issues become PR fires and which do not. Amazon AWS got scrutinized over their failure and its implications for cloud computing, but Microsoft Azure’s failure the same week did not. Similarly, Apple’s location tracking on the device became a PR fire, while Google’s transmission of identifiable (by MAC address) data to their servers from Android phones, did not. (http://www.theregister.co.uk/2011/04/28/google_sued_over_android_location_tracking/) Also, while Apple’s location tracking was specified in that EULA you accepted without reading it, Google’s violated its EULA by not being anonymized.

    Both sides need to take responsibility for the escalation of isolated technical issues into public hysteria. Each side effectively chooses what will become an issue of wide (negative) interest and what will not.

  17. Alistair Croll Sunday, May 1, 2011

    Matthew, I think your core point is absolutely correct. And given that anyone can send a message to the world on today’s Internet—and have that message amplified across social networks—there’s no excuse not to keep customers apprised.

    I do think there are subtle differences to consider, though.
    – In Sony’s case, it was a breach, plain and simple, and the only reason to delay anything was to assist investigators by not playing the company’s hand.
    – In Apple’s case, it “did what it says on the tin”—but the tin was a huge, obscure terms of service document most customers didn’t read, and Jobs’ cursory response only made things worse.
    – In Amazon’s case, I think much of the blame can be laid at the feet of site architects who assumed clouds are like servers. There’s a big difference: as Werner Vogels, Amazon’s CTO, has repeatedly said, in a cloud you design for failure. Most of those affected hadn’t properly designed redundancy. That said, Amazon’s first line of defense against machine failures, known as Availability Zones, didn’t isolate the failure.

    In my experience running large, complex systems, the first diagnosis is seldom the right one, and giving out false information can lead to confusion later on as well as to unwarranted panic that exacerbates the problem. On the other hand, we need greater transparency from the public cloud tools we rely on for processing, entertainment, and nearly every other facet of our lives.

    With most of the legal system firmly entrenched in the world of atoms, it’s going to take crises like these—and the inevitable hand-wringing and lawsuit-flinging—to decide what the right course of action should be. For now, I agree that companies should err on the side of transparency, as long as we allow them to revise their explanations later.

    1. “as long as we allow them to revise their explanations later”
      Does the media ever really do this? My guess is no – The link-bait headlines would then be “Company A Admits It Lied About Problem B” and a new brouhaha would ensue.

  18. This kinds of “need” for instant response even though the accusations may not have been researched properly is the equivalent of so-called journalists demanding Wild West lynching, ie instant “justice”

    What Matt is advocating Ist hat tech repairing be reduced to Gawker like “let’s stir up the angry mobs” sensationalism.

    The problem was not Apple’s response, Matt, the problem is journalist not doing journalism but instead shooting from the hip and crying for blood before any kind of investigation has been done.

    Funny how you cite Toyota as deserving an A+. They took MONTHS to fully research the claims and then the media didn’t believe them. Only a year later when the government confirmed Toyota’s investigation into the cause was driver error did the media apologize for jumping to conclusions.

    Oh wait they didn’t.

    Matt, they used to call this yellow journalism.

    1. Hey Matt – how about this for a compromise:

      When the media starts publishing retractions at the same level of prominence to it’s false statement – e.g. “Sorry we reported X, we were wrong, it’s actually Y”, then companies can start responding quicker.

      I think you’re wrong that quicker responses would reduce the media sensationalism.

      Quicker responses to non-issues would just make it more tempting for the media to drum up scandals where none exist – because a response implies that there was substance.

      Don’t forget that the media has a product too – information. Don’t you think there should be a scandal about how much of it is faulty?

  19. Prof. Peabody Monday, May 2, 2011

    This article reads like a collection of juvenile cliche’s IMO.

    “Apple is a lost cause” because it handles it’s PR so badly? WTF? Based on what evidence? Apple has one of the best PR machines on the planet.

    Just because Apple doesn’t follow *your* idea of how fast a company should respond (based on absolutely nothing BTW), they are in trouble? How many fortune 500 companies have you run?

    Also, these situations that you are crumpling together for this article are in no way similar. Sony has yet to even apologise for their problem (which is a very *real* problem of very *large* proportions), and they didn’t reveal all the information at first when they started talking about it. Apple on the other hand, laid out all the facts out about their issue (which turns out to be nothing but a small bug), in their very first missive.

    The only difference between what Apple did and what you are suggesting is that you wanted them to pick up the phone and say “we are looking into it” first. I think everyone was pretty sure they were looking into it already.

    1. Prof. – Agree. However, Sony’s big wigs DID apologize this weekend.

      Mathew – Why do we need to know stuff before it’s fully ready to be disclosed? Journalists operate in a world of breaking stories. The rest of us don’t live by the same currency you do.

      Last night is a prime example. The reporting on the what, when and where was pretty darned bad leading up to the President’s delivery. Now, is there supposed to be some White House tweet action flowing setting the matter straight in real-time? I don’t think so!! I want THE authoritative read of things ONLY when it’s ready for consumption and not before UNLESS it’s actionable. Hearing there was a tsunami headed my way AFTER the fact… not so good. Hearing there IS a tsunami headed my way early enough to be actionable… Good.

      Amazon’s tsunami hit. No opportunity to adjust course.

      Sony’s network fully pwned. No opportunity to adjust except to lock down your cc/debit card(s).

      Apple’s found to be doing “something” that we’re not quite sure of and they’re lambasted for not divulging things fast enough?
      Next time Gigaom.com goes down… I hope you’re on Twitter, Facebook, IRC, etc giving us the blow by blow of what’s going on and when things can be expected resume. Lead by example Mathew. Lead by example.

      1. Gerald

        Actually we do do that – whenever there is an outage we keep everyone posted on Twitter and Facebook.

Comments have been disabled for this post