CloudFlare Founder and CEO Matthew Prince detailed the problems the company is trying to solve in a blog post earlier this week. In a nutshell, although its network edge that spans 23 data centers is (usually) capable of handling most traditional DDoS attacks, there are a couple types of attacks that target different bottlenecks at the local area network level. In these cases, the 1 Gbps networks ports on CloudFlare’s servers can get overwhelmed, as can the processors themselves.

Of course, when you’re running a multitenant cloud-based service like CloudFlare is, these types of events take on a different urgency:

“Both these problems are annoying if it affects the customer under attack, but it is unacceptable it spills over and affects customers who are not under attack. To ensure that would never happen, we needed to find a way to both increase network capacity and ensure that customer attacks were isolated from one another.”

So, over the course of 2012, CloudFlare spent its time working on what it calls “Project Bondage.” Essentially, that meant configuring the individual ports to look and act like a single port capable of handling much more bandwidth, and then reworking the CloudFlare operating system to prevent external CPU-level attacks from affecting internal workloads.

But the company didn’t stop there. Prince wrote in the blog that CloudFlare’s next-generation servers feature 10 Gbps ports to significantly increase network bandwidth even without port bonding. In an email, he confirmed that rather than use off-the-shelf servers as it has been doing, CloudFlare’s “G4″ servers were designed in tandem with and built by Quanta, the same company that builds Facebook’s servers as well as servers for other large web companies.

CloudFlare still uses off-the-shelf Juniper switches but, Prince added, “[W]e’re tinkering.”

Feature image courtesy of Shutterstock user teflon_timmy.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How tomorrow’s mobile-centric data centers will look
• Web startups: How to guard against security breaches
• What the Data Center World Can — and Can’t — Take from Facebook’s Open Compute Project

The last week has seen probably the largest distributed denial-of-service (DDoS) attack ever. It’s being reported in fairly dramatic terms, with the New York Times and BBC talking about the internet getting jammed or slowed down.

So what’s actually going on? Here’s a rundown of some key points:

A what attack?

DDoS attacks, as the “distributed” part suggests, involve large numbers of computers bombarding a target system with traffic, with the idea being to stop that system from functioning. A bunch of South Korean banks and broadcasters got temporarily crippled by such an attack a week ago, for example.

Who got hit this time?

The intended target appears to be Spamhaus, a European organization that maintains a blacklist of ISPs that supposedly host “spam gangs” and refuse to stop serving them as customers. Spamhaus is pretty resilient, as its own network is distributed across many countries, but the attack was still enough to knock its site offline on March 18.

The reason was the attack’s sheer volume. At the time, it looked to be around 75Gbps of traffic — which is a lot — hammering Spamhaus’s servers. Cloudflare, the security firm that Spamhaus called for help, subsequently published a good explainer of what happened:

“The largest source of attack traffic against Spamhaus came from DNS reflection… [This method has] become the source of the largest Layer 3 DDoS attacks we see (sometimes well exceeding 100Gbps). Open DNS resolvers are quickly becoming the scourge of the Internet and the size of these attacks will only continue to rise until all providers make a concerted effort to close them…

“The basic technique of a DNS reflection attack is to send a request for a large DNS zone file with the source IP address spoofed to be the intended victim to a large number of open DNS resolvers. The resolvers then respond to the request, sending the large DNS zone answer to the intended victim. The attackers’ requests themselves are only a fraction of the size of the responses, meaning the attacker can effectively amplify their attack to many times the size of the bandwidth resources they themselves control.”

Whodunnit?

Spamhaus has no shortage of enemies, given its line of business. Spammers are a nasty lot, although there are in fact some serious arguments to be had around the weight carried by blacklists of this kind, and who controls them.

However, all eyes seem to be on CyberBunker, a Dutch host that prides itself on hosting anything except terrorist material and child pornography (Wikileaks was a client). Spamhaus lists CyberBunker (or CB3ROB, as it is also known) as the world’s number-one offender when it comes to hosting spam gangs, and around 18 months ago it blacklisted the host’s ISP, A2B Internet. A2B responded by reporting Spamhaus to the Dutch police as DoS offenders – if you want to delve deeper into that nasty dispute, here are accounts of what happened from CyberBunker, A2B and Spamhaus.

After this latest attack hit, the NYT got hold of one Sven Olaf Kamphuis, who claimed to represent the attackers. Kamphuis claimed CyberBunker was retaliating against Spamhaus in concert with Eastern European and Russian gangs, saying: “Nobody ever deputized Spamhaus to determine what goes and does not go on the internet… They worked themselves into that position by pretending to fight spam.”

Spamhaus itself is reticent about naming CyberBunker as the culprit. I’ve approached CyberBunker for comment, and will add it in if and when I get it.

UPDATE: CyberBunker has replied with a statement that isn’t a denial of involvement in the attack. Also, I would assume the statement contains significant grammatical errors, otherwise CyberBunker is saying it has never been a recipient (rather than sender) of spam. According to spokesman Jordan Robson: “The only thing we would like to say is that we (including our clients) did not, and never have been, sent any spam. We have no further comment.”

What about this “slowing down the internet” stuff?

Remember that 75Gbps number? Well, that was then and this is now. The BBC quoted Spamhaus CEO Steve Linford on Wednesday as saying the attack had peaked at 300Gbps. That would make it the biggest DDoS in history – or at least the biggest publicly disclosed DDoS.

Professor Alan Woodward of the University of Surrey, one of the UK’s premier computer security experts, told me that the attack “seems to be orders of magnitude larger than anything seen before”:

“In some places it’s been mounted, it has had some collateral damage, for example Netflix, although these are transient effects… The thing that got people talking is that it’s a DNS amplification attack. The point is, if you’re targeting something and [the target has] a 10Gbps switch, you only have to throw 11Gbps at it and you’ve pole-axed the system. If it is at 300Gbps, then potentially some of the main infrastructure is being affected, though I’m not sure how much it’s really affecting it.”

Woodward used the analogy of a highway. Such an attack could briefly take out the highway ramps, he said, but the “main backbone of it is unlikely to be affected for any length of time”.

The thing is, in terms of figuring out whether this attack really has slowed down chunks of the internet, there are other factors to consider. For example, in the last week we’ve also seen a yet another submarine cable cut off Egypt, slowing down internet access in that region. Together, these factors could have a cumulative impact.

“I don’t think there’s any immediate effect on the internet, but it is a wake-up call,” Woodward said. “If it was done really seriously in a wider attack, then it could affect [many users]. Trying to take down the whole internet is impractical, but you could start to decapitate sections of it.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Web startups: How to guard against security breaches
• It’s time for cloud security and big data to come together
• SOPA, OPEN and the fight for the Internet

Affected sites include Wikileaks, 4chan and others according to this Techcrunch report.

One reason CloudFlare opts for Juniper is the latter’s support for the Flowspec protocol which enables customers to propagate router rules across a large number of routers fast, according to the company post. This comes in handy because CloudFlare is always updating rules to combat ever-changing attacks and to re-route traffic as needed to optimize performance.

This morning CloudFlare detected a DDoS attack on one of its customers and its attack profiler ascertained the offending packets were  between 99,971 and 99,985 bytes.

That attack profile was sent out to Flowspec to stop the spread of attacks. From the post mortem:

“Flowspec accepted the rule and relayed it to our edge network. What should have happened is that no packet should have matched that rule because no packet was actually that large. What happened instead is that the routers encountered the rule and then proceeded to consume all their RAM until they crashed.”

Service was restored after about an hour, although CloudFlare said it continues to examine the issue and has contacted Juniper to see if there is a known bug involved or the problem is unique to CloudFlare’s implementation.

Update: On Monday, Juniper said via email that it is looking into the reported network outage.  ”While we have not completed our investigation, we believe this incident was triggered by a product issue that Juniper identified last October, when a patch was also made available. Our customer support team is actively supporting Cloudflare in its efforts to resolve the issue and we are not aware of any other customers experiencing similar issues.”

Cedexis’ Radar view of CloudFlare outage.

Given that the number of DDoS attacks is on the rise, web sites had better gird themselves and hope their security vendors are taking proactive steps to keep ahead of the problem.

This story was updated at 12:25 p.m. PDT with Juniper’s comment.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The promise of SDNs in the enterprise
• The promise of software-defined networking
• Cloud computing infrastructure: 2012 and beyond

Here is GoDaddy interim CEO Scott Wagner’s email message explaining yesterday’s outage:

Yesterday, GoDaddy.com and many of our customers experienced intermittent service outages starting shortly after 10 a.m. PDT. Service was fully restored by 4 p.m. PDT.

The service outage was not caused by external influences. It was not a “hack” and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com.  We have implemented measures to prevent this from occurring again.

At no time was any customer data at risk or were any of our systems compromised.

Throughout our history, we have provided 99.999% uptime in our DNS infrastructure.  This is the level our customers expect from us and the level we expect of ourselves. We have let our customers down and we know it.

We take our business and our customers’ businesses very seriously. We apologize to our customers for these events and thank them for their patience.

- Scott Wagner

  Go Daddy Interim CEO

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How tomorrow’s mobile-centric data centers will look
• Quality of the cloud: best practices for ISVs
• Takeaways from the second quarter in cloud and data

“We had a lot of momentum early in the morning, but now anyone who hasn’t been to our site can’t access it,” Wilcox said. “Our launch got squashed.” Still, he’s philosophical about the experience and says he’s content to stay with GoDaddy depending on what caused the outage and how GoDaddy handles the situation.

And like any entrepreneur, he’s trying to take the lemon of having his site not being up and turning it into lemonade. He’s hoping to find a few other startups whose launches may have been affected by GoDaddy’s outage and see if he can package those up as a story for tomorrow when his site is hopefully back online.

Update: Still working on it, but we're making progress. Some service has already been restored. Stick with us.—
Go Daddy (@GoDaddy) September 10, 2012

And people should check out his project, ThingsWeStart.com (wait until the GoDaddy outage is over), because Wilcox and a team of volunteers has scraped the Kickstarter site for data about all of the projects to determine where they are held and in what industries. (Wilcox says that because he’s not planning to make money off the site, he’s not breaking the Kickstarter terms of service.) Visitors to the site can sign up for notifications of Kickstarter projects in their areas or in certain fields.

And next week, the project hopes to launch a calculator based on more data crunching it’s doing that will tell you things like the best day to post a new project based on your industry, how much you want to raise and other factors. Given how much Kickstarter has become a source of funds for entrepreneurs, having a source of data and analysis on the site’s past projects seems like a good resource. So bookmark it, and check out some of the maps it has released so far. Or check out the file of all the data it has gathered so you can crunch your own numbers.

As for GoDaddy, spokesperson Elizabeth Driscoll said the company is currently working to restore all of the affected services. As to the cause of the outages — rumored to be the work of hacktivist collective Anonymous — Driscoll said, “We have not made a determination.”

With additional reporting by Derrick Harris.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Quality of the cloud: best practices for ISVs
• A near-term outlook for big data
• Infrastructure Q2: Big data and PaaS gain more momentum

The hacking collective is claiming responsibility for levelling a successful distributed denial of service (DDOS) attack on the websites of Virgin Media.

Virgin became the first UK ISP to block its subscribers’ access to The Pirate Bay last week, following a High Court ruling that the Bay breaches record label copyrights and should be blocked by five such providers.

#Anonymous have just taken down #VirginMedia website again because of their involvement in the #Censorship of The Pirate Bay #TPB #OpTPB

— Anonymous UK (@AnonUK) May 8, 2012

But The Pirate Bay does not appear to be on the same page as Anonymous. According to its Facebook page:

“We do NOT encourage these actions. We believe in the open and free internets (sic), where anyone can express their views. Even if we strongly disagree with them and even if they hate us.

“So don’t fight them using their ugly methods. DDOS and blocks are both forms of censorship.”

Virgin Media tells The Register:

“Our website, virginmedia.com, has been the subject of denial of service attacks so we took the site offline for a short period of time. We’re aware some groups are claiming the attacks are a result of the recent High Court order which requires ISPs to prevent access to The Pirate Bay.

“As a responsible ISP, Virgin Media complies with court orders but we strongly believe that tackling the issue of copyright infringement needs compelling legal alternatives, giving consumers access to great content at the right price, to help change consumer behaviour.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social media in Q1: commerce and discovery dominated
• Controversy, courtrooms and the cloud in Q1
• Pinterest reawakens Napster-style debate over copyright

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q4: Platform mania and social commerce shakeout
• Infrastructure Q2: Big data and PaaS gain more momentum

BitTorrent was originally designed with a central server dubbed tracker in mind which would help users interested in the same file find each other to facilitate downloads. However, these tracker servers have become a kind of Achilles heel of the P2P protocol. Once a tracker server goes down, the whole network goes down. BitTorrent programmers came up with a way to discover users without such a server that’s based on the Kademlia DHT technology.

This technology is based on individual BitTorrent clients randomly introducing themselves to each other to establish a kind of distributed directory. However, the presenter, going by the name Astro, showed that one can manipulate some of the data exchanged by BitTorrent clients for trackerless torrenting to introduce oneself to many more clients in the network than necessary and then tell those clients that a popular file is available under a certain IP address.

Astro said that nefarious users could utilize publicly available data from torrent sites like The Pirate Bay to find DHT hashes for some of the most popular files and essentially trick some of these downloaders into attacking a certain target. For example, one could tell tens of thousands of users that an HD version of Inception is available at an address that really is the web server of a corporation. All of these users would immediately try to download the file under that address, bombarding the server with requests and possibly taking it down in the process.

Distributed denial of service (DDOS) attacks were most recently used to take down the sites of major credit card companies as part of the Anonymous revenge for actions taken against WikiLeaks. However, users tend to actively take part in a DDOS attack. In the case of this type of exploit, users may not even be aware that they’re bombarding a bank server with bogus requests while they’re trying to download a movie file.

Astro said that some BitTorrent developers have proposed security measures to prevent this kind of exploit. However, the proposed idea includes a transition period to allow all clients to switch to a more secure version of BitTorrent. Astro said “malicious people” could still use this transition period to initiate DDOS attacks via BitTorrent.

Image courtesy of Flickr user kthypryn.

Related GigaOM Pro Content (subscription required):

• The Quest to Monetize File Sharing
• Are Torrents a Tool for Predicting the Future?
• HTML5’s a Game-Changer for Web Apps

There are two versions of MobileNoter for the iPhone, a cloud version that syncs OneNote to a server and a Wi-Fi version that syncs to any Windows PC directly. I have been testing the Wi-Fi version and while it is a bare bones note taking app it handles my OneNote workbooks quite nicely.

MobileNoter consists of two programs, the iPhone app and the Windows app that runs as a note server. Once installed just tag which OneNote workbooks you want to have available for reference on the iPhone and off you go. Once a sync has been performed on the iPhone side, and this runs quickly over Wi-Fi, all of the OneNote workbooks and note pages are available to view. You cannot edit them on the iPhone within MobileNoter, but they are viewable.

The app does a great job presenting big OneNote pages on the small screen of the iPhone. Tables and complex notes display perfectly, and most importantly to me ink notes look fantastic. Have a look at the screen image included of an ink note in MobileNoter, and you’ll appreciate how well it presents OneNote pages. It’s important to remember that the ink note page displayed was originally a full screen (12-inch) OneNote page.

MobileNoter has basic text note taking capability on the iPhone side. These notes are called Quick Notes and they sync with OneNote back on the Windows PC. It’s a convenient way to get simple notes taken on the fly back to OneNote. The developer has indicated the next version of MobileNoter will allow capturing photos with the iPhone, along with audio recordings to sync back to OneNote. There is also an iPad version in the works, which would be very nice.

Related research on GigaOM Pro (sub req’d):

• How Microsoft Can Win Back the Tablet Market
• How AT&T Will Deal with iPad Data Traffic
• Hot Topic: Apple’s iPad

According to a statement from the phone giant:

Beginning Friday, an AT&T customer was impacted by a denial-of-service attack stemming from IP addresses connected to img.4chan.org. To prevent this attack from disrupting service for the impacted AT&T customer, and to prevent the attack from spreading to impact our other customers, AT&T temporarily blocked access to the IP addresses in question for our customers.  This action was in no way related to the content at img.4chan.org; our focus was on protecting our customers from malicious traffic.

Overnight Sunday, after we determined the denial-of-service threat no longer existed, AT&T removed the block on the IP addresses in question. We will continue to monitor for denial-of-service activity and any malicious traffic to protect our customers.

The story got legs on Reddit, where many commentators scorched AT&T, saying the company “censors the Internet,” and 4Chan fans formed bulletin boards to organize “retaliation” against AT&T. TechCrunch wrote that blocking a web site without notification is an “extreme breach of user trust.” Of course, the blocking evidently wasn’t over 4Chan’s content, which AT&T assures me it would never do, but instead over network protocols that were affecting other customers.

Net neutrality is all well and good, but when one particular server is negatively impacting other users, AT&T is obligated to respond. Not everyone jumped on the blame AT&T bandwagon, however. Late last night, Shon Elliott from unWired Broadband wrote on the North American Network Operators Group mailing list:

There have been a lot of customers on our network who were complaining about ACK scan reports coming from 207.126.64.181. We had no choice but to block that single IP until the attacks let up. It was a decision I made with the gentleman that owns the colo facility currently hosts 4chan [sic]. There was no other way around it. I’m sure AT&T is probably blocking it for the same reason. 4chan has been under attack for over 3 weeks, the attacks filling up an entire GigE. If you want to blame anyone, blame the script kiddies who pull this kind of stunt.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Web startups: How to guard against security breaches
• GigaOM Research highs and lows from CES 2013
• How HR can make the case for workforce analytics