34 Comments

Summary:

The cloud is a wonderful thing, allowing us to backup files to remote servers for just pennies a month. But what happens when something goes awry and your backups start sucking up huge amounts of bandwidth? It’s not pretty, especially when you have a usage cap.

139617711_896179e86e_z

The cloud is a wonderful thing, as you probably know from reading our coverage on the Structure blog and posts from my colleagues Stacey and Derrick — among other things, it allows us all to keep almost unlimited backup copies of our documents, photos, music and other files on a remote server somewhere, and thanks to services such as Amazon’s AWS suite, such backups are pretty cheap as well — pennies a gigabyte. At least, they’re supposed to be. But when the cloud meets an Internet service provider’s bandwidth cap (something that is unfortunately becoming more and more commonplace) it can be a less than happy experience.

I know this all too well. And while some of what happened is my fault, it’s probably not that unusual, so I thought it might be helpful to tell people about it.

Some readers may recall that I wrote recently about my problems with bandwidth usage, and how I thought I had solved them. In a nutshell, my ISP — a Canadian cable and media conglomerate called Rogers Communications — started warning me that my household was using huge amounts of bandwidth, far more than I had ever used before. One particular day last month, the online bandwidth meter showed that we had consumed 75 gigabytes of data, more than three-quarters of our 95-gigabyte allotment for the month.

As I described in my last post, I wracked my brain to try and figure out where this could be coming from. At first, I (and the Rogers technician I spoke to) thought that it was our wireless network, which was unencrypted. So I locked it down with a 64-bit password — but the downloads continued at huge levels, sometimes 20 or 30 gigabytes a day. I interviewed all three of my daughters, aged 13 to 21, and my niece, who is living with us. All denied downloading huge amounts, but one daughter said she had been using a Bit Torrent program for some Japanese TV shows.

I checked every computer in the house to make sure there were no programs running in the background or viruses or malware — we have seven computers in all, including four desktops and three laptops, as well miscellaneous wireless devices like iPhones and a streaming media box connected to the TV. The download usage didn’t even budge. I even borrowed a network switch to try and see which ports on the network were using all this data.

Then a day or two ago, I got an email from Amazon with the bill for my Amazon AWS service. I have about 25 gigabytes of photos, music and other documents backed up to Amazon’s S3 server cloud, which usually costs me about $3 a month — but this time, the bill said $109. Suddenly, a light bulb went on (one that probably should have gone on before then, I admit): that would explain how we could have been downloading 20 or 30 gigabytes a day without our computers overflowing with data. What if something was backing up all that data multiple times from Amazon’s S3 cloud?

I checked JungleDisk, which is the software from Rackspace that I use to manage the S3 instance I rent, but I didn’t have it set to sync or automatically mirror the data. So what could have happened?

After fishing around on the Internet, I finally found a support forum posting at Amazon’s AWS site that described exactly what had happened to me — except it was the user’s mother, and she had somehow download more than 800 gigabytes in a month without realizing it. The culprit, apparently, was either Windows indexing the files or an anti-virus program scanning them, or both. Since JungleDisk maps the Amazon cloud folder as a network drive, Windows and some other programs simply treat it as a regular drive and download all the files to scan them — even multiple times.

Since the amount of data that Amazon says I consumed in April (a little over 600 gigabytes) is almost exactly the same as the amount of extra bandwidth that my ISP says I used in the month, this seems to be the solution. And luckily for me, Rogers has a maximum bandwidth over-usage charge of $50 per month. If it didn’t have this cap on a cap, I would be liable for almost $1,000 based on the company’s per-gigabyte usage fees. So I have to pay $100 to Rogers for overage charges and $100 to Amazon.

A fairly cheap lesson, comparatively speaking, but a lesson nevertheless. And yet another way in which bandwidth caps threaten to turn us all into network administrators and bandwidth cops, whether we like it or not.

Thumbnail photo courtesy of Flickr users Ryan Franklin and Arthur Caranta

You’re subscribed! If you like, you can update your settings

  1. Thanks for this article! Usually I don’t enable the network drive feature of JungleDisk but I did for a couple of clients and could easily have overlooked the AV scanner, although I did disable indexing.

    1. Thanks, Michael — I wish I had too :-)

  2. David Klemke Wednesday, May 4, 2011

    Here in Australia we’ve had data caps for almost as long as we’ve had broadband and whilst I lament having to watch my downloads the tools provided by my ISP (Internode) for monitoring usage are top notch. Of course I wouldn’t expect all ISPs to be that good right after the bandwidth caps come into place, but I don’t think caps are all bad.

    The data caps started out fairly low at first but they’ve been increasing at an exceedingly rapid pace since their implementation. I’ve had my data cap doubled the past year with the price going down by $30, and that’s quite common. I only hope that ISPs in the USA and Canada follow suit since if we can do it, why can’t they?

  3. Wow Mathew, that’s crazy stuff.

    The average Joe may never have figured out this issue… But hopefully I will soon be able to tell you: “Mathew, get a Mac and an iCloud and be done with it. $99 a year – No lessons to learn – Integrated – Just works.”

    Actually, I’m guessing a guy like yourself – massively smart IT-like tinkering type of guy – has no need of something that just works but not enough buttons to control, etc… But that average Joe guy and tens of millions are likely to love the solution. Here’s to hoping…

  4. David mate, tell me why “I don’t think caps are all bad.” Name one good thing about them. They’re just something else to watch out for, as if there aren’t enough – more important – things already. They’re just an attempt to shape the way people interact with media, a last-ditch attempt to save the TV industry.

    Does your ISP provide their top notch tools for Linux, or are you forced to use Windows or Mac OSX?

    1. I think that’s exactly what this is. A chance to capitalize on everything going to the web and trying to save the dying TV subscription model.

      Something has to give. Either they’re going to have to correct downward on TV service pricing, finally offer a la carte plans (unlikely due to back-end licensing), or just give up and compete with each other on BW/per$.

      Unfortunately I think we’re only just entering that painful in-between stage, where it’s finally happening in the US too. I’m waiting for the moment where average-joe netflix user starts dealing with regular overages from his HD streaming, online music library and backup service.

  5. Marc Farnum Rendino Wednesday, May 4, 2011

    > yet another way in which bandwidth caps threaten to turn us all into network administrators and bandwidth cops

    But while it’s convenient to think that infinity is free, there’s no such thing as either, really.

    Shouldn’t we all be a little more cognizant about our consumption of any resource?

  6. The fact that the maximum overage fee is only $50 dollars is enough to convince me that their fee is more about making more money than covering the cost of providing additional bandwidth. What lunacy. The last time I’ve been able to take bandwidth costs seriously is when I connected to the internet through a SLIP connection from a BBS. It makes me wonder if Amazon and Netflix will step up and defend consumers from ISPs trying to take a bigger cut as middle-men.

  7. Is teksavvy not available in your area? Saying goodbye to Rogers is a beautiful feeling.

  8. @David and everyone else from Australia:

    I feel sorry for you but I really wish you would stop posting things like this. Average computer users here in the US don’t understand that the reason you’ve had these caps for so long is that all your bandwidth comes from undersea cables (laid all the way out to your country in the middle of nowhere) which are very expensive and somewhat limited in their capacity. Our politicians here in the US love it when you post things like this because they get their pockets padded by the monopolies who want to confuse customers into thinking they need to implement caps because of expensive equipment or capacity limitations that don’t exist. The truth of the matter is that proposed caps here in the states have nothing to do with the reasons you have them. In the US it is all about profits and control.

    1. All bandwidth has a cost, not just the expensive kind.

      1. Very true Al. And you can take a quick look at the duopolies’ SEC filings to see how much those costs are and whether they are increasing or decreasing. But let me save you the time – bandwidth costs in the US are absurdly low and even with record numbers of people signing up for broadband those costs are decreasing rather than increasing.

  9. Well, that may make the case for caps. Neighbors sharing your cable head might have been quite inconvenienced if their data rate drops while you’re doing a scan that you never intended. I’m not a fan of caps at all, but there is a case to be made when one person dominates a resource to the detriment of others. I think Comcast lowers your rate after 250 gig rather than charge. Seems fine for this Comcast customer. Of course, the great fear is that this will ultimately morph into a means to charge us all more, and force Netflix out of business. Sadly I’m thinking this is the ultimate plan.

  10. A lot of suspense in the story :) But I liked the way it is presented. On a serious note – I think for the kind of scenario you’ve mentioned I need to check a few things as I am using “Dropbox” & Google Cloud Connect & both are auto-sync and may be hogging more bandwidth than expected!

Comments have been disabled for this post