Insufficient bandwidth has the potential to limit the cloud because it can take a long time to send large files over thin pipes. Amazon Web Services now addresses this problem with a new data delivery service called AWS Import/Export, which uses the postal system rather than the Internet to deliver data. Yes, it means customers now have the option to send their data to Amazon’s cloud via the mail.
Werner Vogels, Amazon’s CTO, explains in a blog posting that it would take up to 13 days to sling a terabyte of data across a 10 Mbps network, which is pretty darn slow. So Amazon is offering customers the chance to store their data on an external device, ship it via post, and Amazon will load it into S3. I outlined this problem of needing fat pipes to transfer our increasing loads of data back in April, but was hoping that instead of using FedEx, we’d have faster networks. Interestingly, Vogels doesn’t think our networks will keep up with our data generation — a feeling common also in the supercomputing and cloud storage world. Vogels writes:
No matter how much we have improved our network throughput in the past 10 years, our datasets have grown faster, and this is likely to be a pattern that will only accelerate in the coming years. While network may improve another other of magnitude in throughput, it is certain that data sets will grow two or more orders of magnitude in the same period of time.
I don’t want to believe that faster networks are a lost cause, or that we’ll have to resort to the mail for delivery to the cloud, but as a stopgap measure this offering says a lot about the state of our pipes. For reference, here’s Amazon’s cheat sheet for when it will be faster to turn your packets into a package.
Makes perfect sense.. and in fact long awaited for.. Amazon showing itself as a serious cloud player and offering real solutions, applause!
sneakernet lives! though I guess it’s actually postalnet
mail -> great for first offsite back ups!! after that sysnc changes incrementally using some file sync protocol like rsync
Back in the days of BITnet (anyone remember that?) email would backlog between Canada and the US (we had one link, I think a 56k line to Columbia in NY) and so the mail would be spooled onto tape and trucked to Canada. This is in the 1980s, and I might have the details wrong, but it looks like we’re going back to the old days…
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
What is wrong with sending satellites up in space and do the cloud stuff up there?? Military and scientists already have that… Why not the masses?
You can drive up to a local satellite center to pick up the data..
I might consider moving to the cloud in 2020. Until then I’ll just keep using my faithful terabyte drives.
This was recommended in a UC Berkeley paper (Feb 09) on the economics of cloud computing — see link. Amazon must have just gotten around to reading this paper or finally listened to customers or did a back of the envelope calculation to figure out that snail mail is better than US broadband. Nonetheless, it shows that bandwidth is still a problem in the US.
Maybe Kinkos/FedEx can build high speed fiber connections to Amazon; then customers can drive to Kinkos and upload a terrabyte while making copies of the bill to submit to the boss who will probably yell at the poor worker for not using the Internet to upload the file.
http://berkeleyclouds.blogspot.com/
At work we trade data with other companies. If it takes more than an hour or two to FTP, we consider shipping a disk. If it takes more than a day, we almost always ship a disk.
one thing to keep in mind is that at any network speed over ~1GbE (1000Mbps), the bottleneck is going to shift to writing the media.
eg. for 1TB or higher transfer…
Tmed = Tread = 1hr/500GB
Tship = 16hours (FedEx one day)
Ttotal = 20hours
T(1000Mbps) = 17.5hrs
Tmed = Time to write data to Media
Tship = Time to ship Media
Tread = Time to read/copy Media to cloud server/HD array
Tmed + Tship + Tread >> Tship for large data sets…
Plus the total hourly cost of the operator (intern?) doing the media transfer (for large data sets) is higher than an operator drag/drop dataset to cloud…
Sending data with the postal service is also cheaper than Time Warner Cable’s proposed tiered internet usage plan.
The_5_GIG_Rally.mp3
http://nypostoplobotomy.libsyn.com/
As we used to say back in the day, “Never underestimate the bandwidth of a station wagon full of mag tapes”
Rupert
Back when I got in this business the fastest was to get 40 megabytes cross country was a syquest cartridge and a fedex envelope.
Now fedex is the fastest way to get a terabyte cross country.
Back when I got in this business the fastest way to get 40 megabytes cross country was a syquest cartridge and a fedex envelope.
Now fedex is the fastest way to get a terabyte cross country.
We are so glad Amazon made initial data transfer even simpler. No longer you have to wait weeks before you initial backup completes. Hopefully this will make the tools like CloudBerry Backup for S3 even more appealing.http://cloudberrydrive.com