DSSD was founded in 2010 by Jeff Bonwick and Bill Moore — both part of a select few of engineers with experience building out storage operating systems. With the backing of Bechtolsheim, a Silicon Valley rock star and co-founder of Sun Microsystems, who has backed Google and co-founded switch startup Arista, the company has some of the smartest people in the Valley working there. No one from the company wanted to comment on the story.

My sources tell me the startup is building a new type of chip — they said it’s really a module, not a chip — that combines a small amount of processing power with a lot of densely-packed memory. The module runs a pared-down version of Linux designed for storing information on flash memory, and is aimed at big data and other workloads where reading and writing information to disk bogs down the application.

This fits with the expertise of the team, but this is a problem that others are trying to solve as well with faster and cheaper SSDs and targeted software to to optimize the flow of bits to a database. But the proposal here appears to be about designing an operating system that takes advantage of the difference in Flash memory when compared to hard drives to boost I/O.

For example, on old disk drives you store a group of bits in sequential order, but in reality those bits may get dropped anywhere in the drive. After regular use, when you delete a file, a tombstone marker is placed on the “deleted” file and you have to then find that tombstone and re-write just the amount of data in that space and then find more space for the rest. So the data goes everywhere.

But the DSSD system sounds like it treats files not as a series of bits but as an object that gets a name. That name is the file’s address and it stays the same for the life of the file. The result is there’s no central index that stands between sending the data to storage and storing it, and people can write to it in parallel and not worry abut overwrites. It is both faster and can scale out.

For more details, we can turn to the six patents that DSSD has filed. In mid-March Storage Mojo unearthed patents affiliated with the company that imply it is building a type of faster object-level storage using Flash that’s more durable. From the Storage Mojo article:

So what are they building? They are taking a radically different approach to the problem of high-performance transaction processing storage. The use of flash is a given in TP, and the extra durability, scalability and guaranteed read latency would be very attractive in large TP applications.

The most surprising piece is the object storage-like characteristics suggested by the patents. But handling billions of small objects at high-speed in a flat namespace would make it easy to distribute object indexes among hundreds of users, reducing file system I/O latency. The 3D RAID could eliminate the encoding overhead inherent in advanced erasure codes while providing similar robustness, enabling way-beyond-RAID6 availability.

For those who aren’t storage or computing buffs, the problem here was well explained in a fireside chat that Bechtolsheim had with my colleague Om Malik at our Structure:Data 2011 conference. In it Bechtolsheim outlines the problem that the network causes for access to big data around the 6-minute mark and the need to build new interfaces that can take advantage of the parallelism inside flash chips compared to hard disks. If you do that, you can expand the capabilities of flash beyond just density because you can write data to it faster, meaning the network no longer gums up the works.

Of course, when talking about using flash in more places, there’s always the question of whether this architecture will offer enough of a performance gain to justify the higher price per gigabyte of flash over a hard drive, but for that information we’ll just have to wait.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Dissecting the data: 5 issues for our digital future
• Flash memory: the continuing disruption of enterprise storage

PernixData co-founder and CTO Satyam Vaghani

The company co-founders know from VMware. CEO Poojan Kumar once headed up data product initiatives at the virtualization giant, where he worked on vFabric Data Director and other projects. Co-founder and company CTO Satyam Vaghani was principal engineer at VMware, where he worked on the vSphere kernel and created the company’s VMFS clustered file system.

PernixData’s software, now in beta, will aggregate all of a customer’s server-side flash into one shareable pool. “We make a flash virtualization platform that sits between your apps and the primary storage system and acts as a read/write acceleration tier for all the data in motion,” Vaghani said in an interview.

No rip and replace

The software itself runs on the servers — actually within the hypervisors. It only works with VMware initially but Microsoft Hyper-V and KVM support is promised. That means customers don’t have to replace existing servers or their existing primary storage, and that’s an attractive proposition for many businesses that don’t want to rip out and replace perfectly good hardware.

PernixData claims its software-only approach to this problem is unique. That may be but many other vendors are also attacking the issues associated with server-side flash performance, says Howard Marks, founder and chief scientist of Deep Storage LLC. “There are 22 products in my database that in one way or another use solid state flash to accelerate access to shared storage and make things go faster,” he said.

Making server-side flash virtualization friendly

Legacy giants like HP, NetApp, and EMC, all have their takes on the problem but none of those products fit very well into a virtualized world, analysts said.

“If you put flash in a server, then flash is owned by that server,” Marks said. And that causes problems for companies that want to use Vmotion to move their workloads around. PernixData writes the data to SSDs in at least two servers, which means that if one goes down, the surviving server can flush that data to where it needs to be, said Marks.

Marks does give PernixData credit, which received an unspecified amount of A Series funding from Lightspeed Ventures, for its software-only focus, noting that last year Dell spoke of a similar strategy called Fluid Cache, based on technology acquired from RNA Networks, but it’s not on the market yet.

But as Marks pointed out, there are many other ways to skin the server flash storage cat. “Flash is a complicated space and there are so many startups and technologies it’s hard to see where they all fit. Pernix seems to be the only software-only way to unlock server-side flash to be more scalable but there are many other companies working to enhance flash,” said Stuart Miniman, Wikibon analyst.

 QLogic, for example, bakes software into its adapters that take PCIE server side flash storage or other SSDs and turns them into a SAN device, Miniman said. And, GigaOM’s Stacey Higginbotham outlined Jeda Networks’ take on the problem earlier this week.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data third-quarter 2012
• Takeaways from the second quarter in cloud and data
• Amazon’s DynamoDB: rattling the cloud market

ScaleIO joins other startups that have launched in the latter half of the year hoping to create pools of storage that can be used by any application in a virtualized environment. Those startups include Nutanix and Convergent.io. Fusion-io has a software product in this space as well.

The idea behind this growing number of products is to help address the problems that virtualization has caused for the storage market by making a storage area network (SAN) look more like direct attached storage. In traditional networks someone would configure a server to point to a SAN but these products take advantage of the ever-growing use of SSDs in the data center and overlay software on top to make a more flexible pool of storage. Most of these products, including ScaleIO’s, also let organizations tie in their existing hard disks and SANs into this pool of storage as well.

The result is that IT organizations can take advantage of fast RAM or nearby flash-based SSDs for applications that need a lot of speed. They can also stitch together several nearby SSDs or RAM to scale out as needed while still using existing storage infrastructure comprised of hard drives as well. ScaleIO also touts that its software can move storage resources around even if a node in its pool goes dark like for example, if a systems administrator shuts down the server. That’s a good indication that the team here recognizes that while the virtual servers in a cloud may be ephemeral but underneath it all are physical machines that must be taken into account.

That’s a good understanding to have when you’re building out software that is designed to create an abstraction layer to manage physical devices. ScaleIO’s team has worked at a huge number of scaled-out storage companies including IBM, Topio (acquired by NetApp), NetApp, LSI, and Xtremio (acquired by EMC). The Palo Alto, Calif.-based startup already has 24 pilot customers and five customers in production including SAP and Colt (a Fidelity company).

Feature photo courtesy of Flickr user getinet

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Flash memory: the continuing disruption of enterprise storage
• Report: How Mobile Cloud Computing Will Change Tech
• A near-term outlook for big data

Rackspace CTO John Engates

Unlike its current non-OpenStack storage, this offering lets customers mix and match sizes of block storage volumes as needed with their compute instances. “We used to offer block storage associated with our cloud servers but it came in pre-defined bundles — large servers got a lot of storage, small servers got a small amount. This decouples that decision,” Rackspace CTO John Engates said in an interview.

And, unlike Amazon Rackspace offers both a traditional disk-based tier for $0.15 per GB per month and faster solid state disk (SSD)-based storage, for $0.70 per GB per month.  Snapshots cost an additional $0.10 per GB per month.

Amazon’s analogous  Elastic Block Storage (EBS) is $0.10 per GB/month for provisioned storage for standard volumes,  or $0.125 per GB/month for provisioned IOPS volumes. But Amazon also charges for I/O. Amazon does not offer an SSD EBS tier, although starting this summer it did initiate new “High I/O Quadruple Extra Large” compute instances that use SSD to store and retrieve data fast

Solid-state storage has become an important part of the cloud equation with many vendors. It is very fast, and its coming down in price, but it’s still far more expensive than traditional “spinning disks.”

Rackspace hopes to parlay its OpenStack cloud implementation to challenge Amazon in public cloud and other competitors — including others in the OpenStack camp — in private cloud infrastructure.  A report last week that retail giant Walmart is using Rackspace’s OpenStack cloud to power its big data analysis efforts has buoyed the company. Neither Walmart nor Rackspace will comment on the report which came from William Blair analyst Jim Breen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• A near-term outlook for big data
• Amazon’s DynamoDB: rattling the cloud market

Nimbus Data says Gemini will cut total storage acquisition cost — including software — to $8 per GB from $10 per GB for its current  E-Series product. Company CEO and Founder Thomas Isakovich said one of the chief draws of Gemini, due out in the next 60 days, is its fault-tolerant design that suits it for these mission-critical applications. “This goes beyond the power supplies to non-disruptive capacity expansion, non-disruptive software updates,” he said.

In addition, Gemini’s density should earn it a look by large companies wanting to save data center real estate and energy, he said. “To put it in perspective we’re about 8 times the efficiency of 15K disks. We can fit 1 PB of capacity in a single rack that would take 8 racks and use 10 times the wattage if you used 15K disks,” he said in a recent interview.

This is important as more companies focus “not just on capex but data center opex,” he said, adding: “They’re running out of space and power.” The array also offers software-configurable network interconnects so it can run on Ethernet, Infiniband or Fibre Channel LANs as required.

Analyst Ben Woo, president of New York City-based Neuralytix, said Nimbus Data is in the game to prove that flash is more than a short-term solution. “The fact that they now use interchangeable I/O modules on the back end means companies no longer  have to worry about the networking side, if they change, there’s a new module.”

Priority: proving flash longevity

Vendors like Nimbus have to convince companies that flash can last. Longevity is the “number one risk in an all-flash solution,” Woo said. “Disk drives have a very predictable life, you know when they’ll fail and can do things to mitigate that [but] flash is still a question. Nimbus says they’re taking that to a new level, talking 10 year endurance of their flash,” he said.

Woo also cited the array’s parallel architecture that helps it drive faster throughput.  ”This is an opportunity for them to play against the big boys in media and entertainment who need really fast throughput. Nimbus is doing that through a new set of silicon that helps them drive that performance in a parallel versus a serial manner,” he explained.

South San Francisco-based Nimbus Data already claims some big customers including Bloomberg LP, Wachovia, the State of California,IBM Lucent-Alcatel,  and Stanford University.

Solid-state storage boom continues

Solid state storage is definitely a growing market — researcher IDC expects enterprise flash shipments will grow 20 fold by 2016. But the competition is likewise booming with companies from bootstrapped startups to multi-billion-dollar legacy powers all looking for a piece of that action. Last week alone, startup Skyera emerged from stealth, claiming its new all-flash Skyhawk array will cut native flash costs — before compression and deduplication — to an eye popping $3 per GB. A few days later and at the other end of the spectrum, IBM announced plans to buy Texas Memory Systems, a Houston-based flash pioneer. That deal, the terms of which were not disclosed, is seen as a way for IBM to better compete with storage leader EMC which ponied up a reported $430 million for EXtremeIO earlier this year.

Other entrants include Violin Memory, which just inked  a deal to bring Symantec data management expertise to its arrays, and Pure Storage which recently logged $40 million in Series D funding.

Legacy storage vendors like EMC have touted “tiered” storage — in which companies deploy tape, spinning disks and solid-state or flash memory where each solution is most appropriate. Others like Skyera pretty much tout all-flash-all-the-time.

Nimbus Data’s Isakovich holds a more nuanced view. “Flash can be a lot of places but will not clean the floor. Tape is here, it’s not dying, however we think tiering as it’s done by the big boys is a captive solution and is suboptimal for the user.  We think users will buy fewer tiers because flash is encroaching on high-end disks. People will buy into open tiers, buying best-of-breed for each layer,” he said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data third-quarter 2012
• Flash memory: the continuing disruption of enterprise storage
• Takeaways from the second quarter in cloud and data

Solid-state, or “flash”, storage is hot, hot, hot, hot. A few proof points: Pure Storage just netted $40 million in Series D funding and storage kingpin EMC bought XtremeIO for $430 million last May.

But Skyera claims it’s out-doing Pure Storage, Violin Memory or Nimbus Data and others in making solid-state storage more price competitive with as inexpensive as hard drives and by resolving longevity issues that make IT buyers nervous about adopting the technology. The company claims it can cram 44 TB of storage in a 1u rack that comprises a ground-up-engineered system including the flash controller, RAID controller, storage blades and network interface.

Taking flash mainstream, really

“Competitors got their [native] cost to $7 or $8 per gig and that puts you at the high-end niche level of enterprise applications where IT has to put such a premium on performance they’re willing to pay ten times more than a comparable hard disk system,” Skyera sales VP Tony Barbagallo told me in an interview. “Solid-state storage will not go mainstream at those prices and our goal is mainstream.”

Skyera’s founder and CEO is Rado Danilak, who founded SandForce, a storage technology company LSI bought last year for $400 million. He and his engineering team has a ton of credibility in this arena. SandForce controllers own about 80 percent of the market, according to some estimates.

To get to cheap flash enterprise storage, Skyera uses inexpensive Multi-Level Cell (MLC) NAND flash, but found a way to do it to prolong the lifespan of the media.  It is able to use high-density sub-20-nm MLC flash because its controller dynamically adjusts as the medium ages to reduce damage over time.

Putting flash everywhere

The use of consumer-grade flash in enterprise storage is not new.   ”What’s cool here is how much capacity [Skyera] gets in a very small footprint and that they’ve integrated the switching capability right into their array,” said Jim Bagley, senior analyst with Storage Strategies Now, a storage consultancy.

The Skyera team appears to have come up with the next generation of that popular SandForce controller and crafted a huge performance improvement in the process.

Skyera will demonstrate its Skyhawk systems at next week’s Flash Memory Summit. Expect a spate of other flash-related news to come at that event and at VMworld, the following week.

Most storage vendors still talk about using flash judiciously in the enterprise, reserving it for jobs that demand the fastest response. Flash often acts as a “turbo” to an existing storage engine, said Mark Peters, senior analyst with Enterprise Strategy Group.  “Skyera is saying:  ’forget that, use solid state for everything.’ The only reason we haven’t done this already is money. It all comes down to price. And, if Skyera has done what it says it’s done on price, this truly is disruptive,” he said.

Fusion-io, a maker of in-server flash, stands at about $11 per GB, Nimbus Data, a Skyera competitor, at about $10 per GB s0 if Skyera can deliver its promised $3 per GB target, it’s a pretty significant cut.

George Crump, lead analyst for Storage Switzerland agreed. “I get asked all the time when flash will take over the data center and the answer is: when it’s the same cost as hard drives.”

If Skyera Skyhawk  lives up to its billing, this is a big step toward that goal. “The next question is if or when they’ll get leapfrogged? We’ll see,” Crump said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data third-quarter 2012
• Flash memory: the continuing disruption of enterprise storage
• Takeaways from the second quarter in cloud and data

As described in a post to the the AWS blog late Wednesday, each “hi1.4xlarge” instance can access two 1 TB volumes of fast solid state disk (SSD) storage to enable that speed.

According to the blog:

The SSD storage is local to the instance. Using PVM virtualization, you can expect 120,000 random read IOPS (Input/Output Operations Per Second) and between 10,000 and 85,000 random write IOPS, both with 4K blocks. For HVM and Windows AMIs, you can expect 90,000 random read IOPS and 9,000 to 75,000 random write IOPS. By way of comparison, a high-performance disk drive spinning at 15,000 RPM will deliver 175 to 210 IOPS.

Netflix, a huge Amazon customer, has already benchmarked the new SSD-enabled instances running a Cassandra test suite. Adrian Cockcroft posted a blog outlining the benchmark is here, but the short version is that Netflix validated Amazon’s raw performance claims for this Cassandra workload. Cockroft is director of architecture for Netflix.

The availability of solid state storage, which is faster and more expensive than more traditional spinning disks, has become a battleground with more companies adding the option as the price for SSDs falls.

Update: One Amazon user, Robert Shears, president of Greystone Solutions, a Boston developer of e-commerce sites, is generally impressed with the speed at which Amazon makes new features available. This news is more exciting than usual, he said, even though only a few of his customers really need this capability. “For those who do need it, it will be a real lifesaver,” he wrote.

“Our expected use cases are first: SQL Servers that need to scale up in a big hurry due to volume demands and second: clients who are stuck with legacy DBMS systems (relational or pre-relational) who have important line-of-business apps that are running out of steam at the same time DBMS vendors are also running out of steam.”

The specs for the new instances, which initially will be available from Amazon’s US East in Northern Virginia and EU West regions, are:

• 8 virtual cores, clocking in at a total of 35 ECU (EC2 Compute Units)
• HVM and PVM virtualization
• 60.5 GB of RAM
• 10 Gigabit Ethernet connectivity with support for cluster placement groups
• 2 TB of local SSD-backed storage, visible to you as a pair of 1 TB volumes

Early comments to the post were positive. Wrote one commenter:

Excellent! But please consider adding a wider range of instance types with SSD storage. Not every server needs 60GB ram and 1TB ssd. Ideally every current instance type should be available with the choice of SSD storage instead(or at least a selection of the most popular ones).

Oh and it would be very nice being able to spin up some Amazon RDS MySQL instances backed by SSDs.

Amazon continues to add new features and functionality to EC2 at a fast clip. It will probably have to keep doing so as Microsoft and Google and other well-funded players appear intent on challenging its dominance in massive cloud infrastructure.

For more on the new high I/O instances, check out the video.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises
• AWS Storage Gateway jolts cloud-storage ecosystem

Here are some takeaways:

1. Private cloud mandate remains, but changes: For most financial services firms, secure private clouds (i.e., clouds they build and manage in house for themselves) remain the way to go. But Spiers foresees possible change. Over the past six months, he’s seen evidence that vendors are starting to “get” data security requirements and is more confident that the bank will build a secure private cloud with vendor partners. Over time, that is.
2. GPUs are big in banking: The BofA uses GPU (graphics processing unit) computing techniques often seen in online gaming and in high-performance scientific computing to run simulations in its derivatives business.
3. Applications have to catch up with cool technologies: Solid-state storage and in-memory databases are great if you have software that can use them. But … “I want software that I can buy that takes advantage of storage hierarchy changes. SSDs are a great first step in boosting performance, but applications are not yet written to take advantage of this technology,” Spiers wrote. Similarly, despite talk about “benefits of in-memory database as the network hierarchy expands with large, fast options, the software is not optimized at the drive and OS level,” he said.
4. Other hot technologies worth watching: Non-volatile memory, software-defined networking and more parallel compute options have all caught his eye.
5. What BofA runs now: As of the end of last year, Bof A ran just south of 90,000 physical servers and 40 percent of them were running a hypervisor. Most of these multi-core, mid-range machines have 16 to 96 G of memory, and run Linux and Windows. Many of the bank’s businesses were running in IT “silos.”
6. What’s coming: A standards-based compute stack based on X86 servers, 10GbE networking, 16 GB Fibre Channel and NAS storage; Windows and Linux OSes and automation tools. The goal is to consolidate a lot of the workloads on the same infrastructure,  boost speed to market, allow elastic AWS-type scaling and better resilience and availability.

Photo courtesy of Flickr user Alex E. Proimos.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Public, private or hybrid? How to move to the cloud
• Forecasting the future cloud computing market
• A near-term outlook for big data

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing infrastructure: 2012 and beyond

AWS doing solid-state drives is a big deal in the world of cloud computing, where users have been wondering for years when the company might start offering SSDs as a service. Other cloud providers already offer bare SSDs as a service, and more certainly are thinking about it with the advent of companies such as SolidFire that are specifically targeting cloud providers with solid-state arrays. The idea is that they’ll be necessary to run I/O-intensive applications such as databases and ERP, which many large organizations consider mission-critical but which many cloud providers aren’t yet equipped to handle.

In that sense, DynamoDB is something of a curveball. It lets AWS users leverage the performance of SSDs, only as the underpinning of a new service rather than as a new IaaS feature alone. But that’s also what makes Amazon’s decision so wise. By launching a new service that lets developers experience SSDs in a fully managed service so they don’t have to jump through the hoops of building, deploying and managing an SSD-based cloud application, AWS distinguishes itself from the pack, especially among the web developers that are its bread-and-butter use base.

Web developers use NoSQL databases more frequently than enterprise developers, and NoSQL requires solid-state performance. As several MongoDB users explained at last month’s MongoSV conference, using AWS’s cloud means trading hard-disk performance in exchange for convenience and scalability. That trade-off, however, means data must be stored in-memory rather than on disk, which means higher operational costs. At $1 per gigabyte per month, DynamoDB provides SSD performance at a price point far lower than relying on the native memory of EC2 instances (at least in terms of capacity costs).

It’s services, rather than just Infrastructure-as-a-Service, that most live up to the promise of cloud computing. Going forward, one can envision AWS using SSDs as the basis of numerous other services, including its rumored analytics service and maybe even for a speedier version of its Relational Database Service. Look at Elastic MapReduce. Anyone could use Amazon EC2 instances to build a virtual Hadoop cluster, but Elastic MapReduce eliminates the hassle of even managing that virtual cluster.

It seems likely AWS will eventually offer access to bare SSDs as a service for developers who still prefer to build applications using non-Amazon pieces and who will probably always represent the majority of developers. But that’s a relatively low-margin business for AWS, and a relatively low-value proposition for users. Web developers, enterprises, third-party cloud database providers, data scientists, whomever, will always need access to cloud-based incarnations of the latest-and-greatest hardware, but cloud providers that really want to stand out will always have to look higher.

Image courtesy of Flickr user shimelle.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Amazon’s DynamoDB: rattling the cloud market
• The fourth quarter of 2012 in cloud
• Understanding and managing the cost of the cloud