David Yach, the CTO of software for RIM (NSDQ: RIMM), today said that the ongoing service outage that is affecting millions of BlackBerry users — consumer and enterprise — is not down to ongoing system failures, but to a backlog of messages that built up when the company’s servers in Slough, England, failed on Monday. In a very brief conference call today providing an update on the situation, he said that global teams are “working around the clock” to get things back to normal, but also admitted that the company was still not completely certain of what it was that went wrong in the first place.
“We believe we have identified the case of the original failure,” he said in the call, but engineers are still investigating the issue and making a “deeper analysis” of the situation. He noted that the company has so far seen “no evidence that a hack is the reason for this.”
Some of the highlights of today’s call:
How did the problem spread? Yach’s answer for this was simple: backlog of messages. Apparently, the build-up of messages from territories outside of those covered by the NOC in Slough, into the Slough territories that were not able to be delivered, has created a backlog of messages that are now slowing down the system.
Is that all? Not exactly. It seems that in Slough RIM has actually actively slowed down transmission of messages to assess the problem (something that Rory O’Reilly, an EMEA VP for Software, described to me this morning). But Yach notes “at this point we have not throttled the other regions.” The slowdown elsewhere is that messages are being held up . The network operations center in Slough, England, covers the EMEA region but also some territories in Latin America and Asia.
Although millions of people are without service, it’s important to note again that this is not a blanket issue. The BlackBerry Internet Server runs the BBM messaging service, emails and all Internet data services. Some users are seeing only intermittent problems; some are seeing problems only in one data service but not another; and some see no problems at all.
Will any messages get lost? “We will not be dropping any email messages,” said Yach.
What about compensation for those affected? Users will still have to wait to hear more on that — even as they continue to slam RIM in forums like Facebook and Twitter, and the stock falls. “At this time we’re just concentrating on getting the service running,” said Yach, who offered no comment on any efforts to compensate users for any loss of service.
The fallover in RIM’s BlackBerry services is a double-whammy of sorts. The impact on services like BBM, very popular with consumers, gives the service failure a much higher profile than it would have had if it had been restricted to enterprises.
But the fact that the outage is also affecting enterprises means that it is hitting at RIM’s perhaps less vocal, but extremely valuable core customer base. RIM has prided itself on offering the most resilient and robust enterprise mobile services. That image will have been severely impacted by events of the last few days, especially since RIM has not been able to fix the problem — which has only extended in terms of the geographies being impacted as the hours have passed.