I recently happened upon Apple’s classic “1984” commercial . It had been years since I’d last seen the ad, but I was struck by its symbolism and timelessness all the same.

The ad, which features a runner heaving a hammer into a giant monitor on which a Big Brother-like figure is speaking ominously to a room full of drone-like workers, signifies a classic case of technological disruption. Apple, playing the disruptor, was introducing the Macintosh and personal computing to the market and forever changing the face of computing in the process.

Since the commercial aired, computing technology has evolved at an astronomical rate. Personal computing spurred the rise of network computing, which in turn has spurred the rise of modern technologies like virtualization and cloud computing.

While the computing side of history is well known, the storage side remains hidden from common view, like the bulk of an iceberg. It’s interesting to look at the state of storage today and compare it with the radically different environment that existed back in 1984, during the rise of the personal computer.

NFS: The dawn of the modern (storage) era

Storage as it is known today did not even come into being until the mid-1980s, when Sun Microsystems introduced the Network File System (NFS) protocol. Before NFS, servers simply consisted of direct-attached disks that connected to a general-purpose computer.

NFS was a huge step that both enabled and accelerated network computing. While it was initially met with a healthy dose of skepticism, NFS quickly gained traction in the enterprise. As it became increasingly clear that computers needed to actively share information and interact with each other, networked file systems became a central tenet of storage.

Just like that, the next generation of storage architecture was born.

During this time, I was a Ph.D. student in the computer science department at UC Berkeley, working with a talented team to perfect RAID, which is the basis of today’s multi-billion dollar storage industry. Vendors offering purpose-built systems for managing storage based on RAID technology — notably EMC and NetApp —grew rapidly. As enterprises embraced network computing, purpose-built storage systems based on RAID were quickly recognized as necessities, rather than niche products.

Virtualization demands more than general-purpose storage can offer

Twenty years later, we are in the midst of the single most significant evolution in IT since the rise of network computing: the rise of virtualized computing. While virtualization offers unprecedented benefits to the server side, including server consolidation, agility, flexibility and manageability, the full-scale adoption of virtualization has been stalled by the complexity and cost of storage. Today, storage is the single most complex and expensive component in the virtualized data center. As Tintri CEO Kieran Harty noted in a recent blog post, storage can account for up to 60 percent of the cost of virtualization deployments.

In fact, it is becoming increasingly clear that general-purpose storage is not sufficient for supporting broad virtual deployments due to fundamental limitations in its architecture. Because general-purpose storage must support a wide range of workloads, it is inefficient and difficult to configure and manage for virtualized environments. With these systems, it is difficult to see what is happening at the level of individual VMs, and you cannot perform storage management operations at the level of VMs. General-purpose systems are difficult to automate for use with virtualized environments.

Moreover, existing general-purpose storage systems were designed before the advent of new technologies such as flash-based SSDs, multicore CPUs, and 10 Gigabit Ethernet. Although most provide support for such technologies, their legacy architectures cannot take full advantage of them.

Storage is about to be disrupted

This has created a disruptive opportunity in the storage field that hasn’t been seen in over two decades. It’s not a surprise that there’s been such a flurry of venture-backed startups entering the market over the past year. Entrepreneurs — many of them coming from established tech giants — are capitalizing on technologies like flash to introduce more efficient storage solutions for today’s data center.

In the virtualized data centers 20 years from today, all aspects of computing will be virtualized, including servers, networks and storage. Virtual machines will be freed from the constraints of the underlying physical resources and will run with the same level of functionality and service, wherever it is most efficient. Full virtualization will be achieved, not only by an accumulation of new features, but by designing an architecture that eliminates everything that cannot be efficiently virtualized. Just as network computing spurred the need for an entirely new storage architecture, so too has virtualization.

Ed Lee is lead architect at Tintri, Inc., which has developed a storage system for virtual machines. Prior to Tintri, Ed was principal systems architect at Data Domain.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Smartphones help us to understand the cloud
• For UK education, private clouds may make economic sense

LaCie’s desktop RAID drives feature either two 500 GB 7200 RPM or two 1 TB 5400 RPM hard disk drives mounted in a single case, and each features two Thunderbolt connectors for attaching to your computer and for daisy-chaining with other Thunderbolt-equipped accessories. Mini DisplayPort monitors can also connect to the LaCie Big Disk Thunderbolt series, but one of those must cap the daisy chain if you plan to use it with other Thunderbolt devices.

The drives are light, at 1.4 lbs each, and can achieve transfer rates of 251 MB/s. If you daisy chain four together, LaCie says you can hit even higher speeds, up to a maximum of 782 MB/s. Note that this would also require four Thunderbolt cables, which Apple sells for $49 each. And the LaCie drives don’t ship with a Thunderbolt cable .

LaCie’s target market with these drives is creative professionals who needs need the speed of Thunderbolt and also a conveniently portable form factor. The Little Big Disk is a much cheaper option than the Promise Pegasus RAID System, which debuted in June and starts at $999 for a 4 TB version. Still, it probably won’t appeal to the average consumer for whom an equivalent 1 TB drive at around $70 or a little more with USB or FireWire connectivity is more than adequate.

If I did just a bit more video and photo editing than I do now, I can see the LaCie being a good option for an external project scratch disk. LaCie also supposedly has an SSD option on the way, but that will likely be even more expensive. Anyone else thinking about grabbing one of these?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• New challenges for the IT organization
• Mobile Q2: Smartphone growth surges; iPad’s rule continues
• Why iPad 2 Will Lead Consumers Into the Post-PC Era

Each cable has a controller at either end, which is used to regulate the speed of data transfer and boost the signal to make Thunderbolt’s extremely low-latency transmission possible. Each end contains 6 chips, including 2 Gennum GN2033s and 4 smaller ones, making for a total of 12 chips in each cable. This makes the Thunderbolt cable an “active” cable that has its own internal firmware and allows it to manage the high two-channel independent 10 Gbps transmission speeds.

But it also isn’t cheap. And for right now, Apple is the only game in town when it comes to Thunderbolt cable suppliers. As Ars Technica points out, the situation bears some similarity to the early days of FireWire, which was initially very costly and limited to Apple because of unfavorable costs when compared to USB. Apple didn’t help things by initially requiring licensing fees for the use of the FireWire trademark and logo. Ars argues that the similar high costs of Thunderbolt could limit its ability to gain a real foothold.

I see the similarities between Thunderbolt and FireWire, but I think it’s too early to assign them the same ultimate fate. For one thing, Apple is well aware of how the FireWire situation panned out. The Mac maker isn’t likely to repeat the same missteps with Thunderbolt if it really does intend for the tech to have wide applicability. Second, Thunderbolt is like FireWire, but they also can’t really be compared in terms of what they allow a user to potentially do. Display connectivity, along with speeds that basically allow Thunderbolt to act as an external PCI connector, give it a much broader scope in terms of applicability. Want to set up a server with upwards of 50 terabytes of storage running through a Mac mini? Easy (and relatively cheap using the new Pegasus RAID drives), once an updated Mac hits the market. Plus there’s always the scenario of the computer-on-a-drive that can be booted from any Mac, apps, files and settings intact.

Apple is also in a much different position than it was when it introduced FireWire. Its share of the PC market has never been stronger, and it continues to experience growth in that sector. It also has a huge chunk of the rapidly expanding mobile industry, thanks to the iPhone, iPod and iPad. While Thunderbolt tech hasn’t yet made an appearance on the mobile side of Apple’s business, I think it’s only a matter of time before it does. And then it doesn’t matter who else embraces it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Who and what to watch in the new era of the living room
• Analyzing the wearable computing market
• Monetizing music in the post-scarcity age

Until today, it wasn’t clear whether users could boot from an OS X install on an external drive attached via the new high-speed Thunderbolt data transfer specification, as is possible using FireWire. AnandTech (via MacRumors) has already got a new Promise Pegasus 12 TB RAID system with Thunderbolt, however, and they’ve found that booting over Thunderbolt is indeed supported.

That means that you could run an entire OS X install, complete with your apps, files and preferences, on an external Thunderbolt drive, and then unplug said drive and take it with you wherever you can find another Thunderbolt-equipped Mac. In theory, working with an SSD drive attached via Thunderbolt should feel much faster than working with even a 7200 RPM HDD installed inside your MacBook, for example. Depending on how pricing of third-party external Thunderbolt drives goes, you might even see users buying the minimum onboard storage for Macs and just booting every time from a much speedier or more capacious desktop drive.

This will probably have the biggest impact for mobile workers and Macs in the enterprise. Employers could use Thunderbolt storage to make workstations hot-swappable, allowing them to shift around staff to different machines in different offices or departments as needed. Mobile workers might be enticed to use coworking temporary office facilities that rent Thunderbolt-equipped Macs by the hour, day or month instead of buying high-end gear that they only use sporadically.

Consider also that adding a RAID card and 8 TB of storage to a Mac Pro costs $1750 before tax using Apple’s customization options at the time of purchase. An external Promise Pegasus Thunderbolt RAID system boasting the same storage costs only $1500, and that price will likely drop as Thunderbolt costs drop and more competitors enter the market. Thunderbolt could help make professional-caliber rigs more affordable for prosumers and consumers.

If you thought $50 was expensive for a single cable, this feature alone makes it worthwhile in my opinion. What do you think?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the iPad is Right for the Enterprise
• Who and what to watch in the new era of the living room
• New challenges for the IT organization

Admittedly there aren’t many peripherals that use Thunderbolt, or many Macs that have Thunderbolt ports, but as Apple gears up for a whole slew of new Thunderbolt-enabled computers, a bunch of new peripherals are also in the pipeline. And in order to tap that opportunity, Apple introduced a new Thunderbolt cable today, available in the online store for $49.

In the product description for the new cable, Apple talks about the Thunderbolt’s ability to transfer data using two channels of 10Gbit/s each, and how it lets you use peripherals like hard drives. But the $999 Promise Pegasus 4×1 TB RAID drive, also now available for sale in the Apple online store, is currently the cheapest way to start using Thunderbolt storage. Other Pegasus RAID arrays are available in configurations ranging from 6×1 TB to 6×2 TB, costing up to $2,000.

It’s no surprise that both the cable itself and the Pegasus drives have such a high starting price point. Thunderbolt always seemed destined to appeal to professional users first, since the new technology will require more time on the market and a wider reach before it can bring production costs down. Apple seems to be working on expanding the potential pool of Thunderbolt users, however, as reports suggest that we’ll see new Mac minis, Mac Pros and MacBook Airs with Thunderbolt in the very near future.

With all those Thunderbolt Macs around, the new cable will become even more useful, as it also provides users a simple way to establish a very high-speed connection between computers using Target Disk Mode. Plus, you can use the cable to connect a Thunderbolt-equipped MacBook Pro to the latest iMac in order to use the iMac’s screen as a second display. Those features alone might be worth $50 to some multi-Mac consumer households.

If you own Thunderbolt-equipped Mac, are you planning on getting one of these cables, or will you wait until there’s more third-party gear available to use it with?

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Tablet market to hit over 377 million units by 2016
• How do developers ride the Siri wave?
• Updated: Forecast: global mobile subscribers, 2010-2015

Other World Computing provided a review unit for me to test. The company sent it to me pre-configured, so I just plugged it into my 13″ MacBook Pro, and it was ready to go. I connected the drive via FireWire 800, but the drive also includes eSATA, Firewire 400, and USB 2.0 ports, so it’s compatible with pretty much any Mac or PC system.

It’s available in a variety of sizes from 640GB to 2TB, and you can choose standard SATA or SSD drives. You can even provide your own drives if desired.

In addition, the drive features fanless operation, an aluminum enclosure to dissipate heat, a shock isolation system, and there’s no need for a power cord when it’s connected via FireWire. An optional power adapter is needed when connecting via eSATA.

I found the drive’s performance to be excellent and very quiet. The manufacturer says that maximum data transfer rate varies based on the method of connection:

• eSATA: 1.5 Gbit/s (or 300MB/sec)
• FireWire 800: 800Mbps (or 100MB/sec)
• FireWire 400: 400Mbps (or 50MB/sec)
• USB 2.0: 480Mbps (or 60MB/sec)
• USB 1.1: 12Mbps (or 1.5MB/sec)

It’s reasonably-sized at 5.6″(D) x 6.1″(W) x 1.1″(H). And it’s sturdily built, too; in fact, it feels so sturdy that it’s a bit on the heavy side.

The Pro Dual mini is priced at $179.99 to $3,199.99, depending on configuration. It comes with a three-year warranty.

If you need convenience with the extra security of RAID while on the road, the OWC Mercury Elite-AL Pro Dual mini is worth looking into.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Millennials in the enterprise, part 1: strategies for supporting the new digital workforce
• The Future of Work Platforms: An Overview
• Social Media in the Enterprise

Active Storage was born out of Apple’s decision to discontinue the Xserve RAID. Alex Grossman, CEO of Active Storage, and the other founders were previously executives in the Servers and Storage products group at Apple, Inc. They left to form their own company that would produce storage solutions of the same quality. While Apple has been pitching the Promise VTrack RAID as a replacement solution, Active Storage has developed its XRAID product line to be a lot more Mac-like with brilliant Mac-native management software. This is high-end gear for serious business — fibre-channel, redundant controllers, redundant power supplies, redundant cooling, etc — with an emphasis on performance. The XRAID comes in two flavors, the original XRAID with 16TB capacity expandable to 32TB and the XRAID ES with 4TB, expandable to 16TB.

All of this power is wrapped in a sexy package that feels very Apple like. The folks at Active Storage have put a lot of care into building enclosures with no sharp edges, tapered screws that sit flush with the chassis, thick gauge aluminum, and slick handles on the drive sleds that pop out with a light touch. I had a chance to take a look at the XRAID at Macworld Expo in January and the build quality on these units is astounding. Having been in a data center or two in my life, finding a polished product where attention has been given to every detail to make it simple to access from both the front and the back of the rack is a real joy. One of the coolest pieces is an iPhone monitoring app that gives you all the details you could want on how the XRAID is performing even when you are not near the data center. This is definitely a storage solution fit for a Mac.

The extreme ease of use in a Mac environment has made Active Storage popular in the entertainment crowd. When asked to explain further, Mr. Grossman explained that, “You may not know this because you aren’t in working with these systems every day, but feature-length movie production requires 50 to 100TB of storage. And that need doesn’t go away when the movie is finished either. There is a need for long-term archival storage as well. And it doesn’t always make sense to use tape backup, so disk-to-disk archiving is growing.”

Final Cut has made the Mac platform a common choice for film editing with some feature films like King Kong being composited and edited entirely on Macs. With the advent of HD, the storage needs for films are growing astronomically. Active Storage appears to be positioned to take advantage of that trend. It makes a great hardware solution in conjunction with Apple products like the Xserve, Final Cut Server, and Xsan 2.

The company currently has about 25 employees, but about 50 open positions on its web site. It is obviously looking to grow substantially in 2010, especially with this capital available to fund hiring and product development.

It is great news to see a well-respected investment fund like Intel Capital place bets on the Mac market, especially in a company that is focused on enterprise-class solutions for the Mac.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Mobile Q2: Smartphone growth surges; iPad’s rule continues
• TV Apps: Evolution from Novelty to Mainstream
• The fourth quarter of 2012 in cleantech

I was recently introduced to the Drobo box. If you’re not familiar with it, simply put, it’s the ultimate home backup server.

For $400, you get a BeyondRAID array that can support up to 4 simultaneous drives, and caps at 16TB. Even cooler is the ability to swap out your drives with a single push of a button. Drobo will backup your files across each drive, so if one fails, your files are still safely stored on the others.

Backing Up

Drobo makes it apparent how easily it integrates with Time Machine right from the start. You can set the limit to how much Time Machine will backup on your Drobo drives, and with FireWire 800, or via your network, it’s hassle free backup just like Time Capsule.

iTunes Streaming

To add a little envy to the product, DroboApps features an iTunes Media extension that allows you to share your iTunes library across your network. Keep in mind it costs extra for the FireWire and Network support as it’s a separate attachment. So not only will it back up your data, but now it can stream your iTunes content to your Apple TV or computers across your network.

The Competition

Drobo may offer redundancy and media sharing,  but let’s look at some other viable options. Correct me if I’m wrong, but if you were to hook up a secondary drive to your Apple Time Capsule as a NAS drive, wouldn’t it be the same thing? One drive for back up, and one for sharing media across your network?

I question Drobo, only because from the people I know who have experienced it, and based on Drobo’s Amazon customer reviews, the device doesn’t seem 100 percent reliable. However their customer service record looks to make up for it.

Also considering other alternatives, such as HP’s Media Smart Server, I’m left wondering which is the most effective, but ultimately cost effective device for my home. I guess it really comes down to what matters most to you. If you’ve had experience with any or all of these devices, please share in the comments.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• NewNet Q4: Platform mania and social commerce shakeout
• The Backup Barrier: Obstacles to Online Storage Strategies