In a conversation after our discussion onstage at Structure:Europe on Wednesday, Parikh said that instead of just using flash in places where you need performance, “like adding a Ferrari engine to your server,” the social network was thinking about other places it might be useful. He declined to go into more details but implied there were places one could replace hard drives with flash. Flash is common in data centers where performance matters, such as for database storage and for speeding up access to images. Facebook uses flash from Fusion-io is some areas of its operations where performance is essential.

But flash is expensive when compared to the traditional hard drives that most companies use for storage. Part of that expense comes from the work that goes into making flash memory — which was designed for consumers — dependable enough to handle enterprise data. Flash can be unreliable, and as its storage capacity grows, the number of times it saves or accesses data shrinks. And while the cost of flash is dropping, hard drives are still cheaper on a per byte basis.

But flash memory is also more energy-efficient when compared to hard drives. And several sources have told me that Facebook is building software that will allow it to take advantage of flash that has been rejected for high-performance use cases like memcached servers and instead use it for storing data that might otherwise go on hard drives. The software would map the unreliable areas of the flash memory and then avoid putting data in those spots. Other sources have mentioned that Facebook is building “flash appliances” which could be arrays of the rejected flash drives and the software to manage them.

And I think that kind of appliance or custom-flash array would be perfect for Facebook’s cold storage facility it has been talking up recently. The idea behind the cold-storage facility is that Facebook wants a place to efficiently and cheaply store those 220 billion photos it has — it gets 7 petabytes of new photos each month, Parikh said during our on-stage conversation at Structure.

Since after a few days the desire to access those photos diminishes, Facebook needs a place to put them that’s cheap but still accessible for those times users are feeling nostalgic. Cheap flash-based arrays would be perfect. Because the data in cold storage is infrequently accessed, the diminished read-write capability of the drives is less of a problem. Because flash has no moving hardware it’s not only greener, but its something that can be turned on instantly. In an interview with Wired Tom Furlong of Facebook said the equipment in the cold storage array would be designed to turn on and off when not in use.

So, Facebook officially says it wants to use flash in more places, while sources have been sharing details of Facebook’s flash appliances. I think that adds up to flash being a key component of Facebook’s cold-storage boxes. I reached out to Facebook for this story, but have not heard back.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang
• Finding the Value in Social Media Data

Ahead of his talk with me later this month at our Structure:Europe conference in Amsterdam, I spoke with Jay Parikh, VP of infrastructure engineering at Facebook, about the computing challenges facing the giant social network. The one most on his mind at the moment is how to store users’ photos, videos and other digital bits so they can access them anytime they want. Like the piles of albums I have from my high school days, our digital photos have to live somewhere, so Facebook is trying to create a data center equivalent to that dusty old box in the attic that you only open when you move.

Jay Parikh

He says Facebook is rethinking the infrastructure for how it stores huge repositories or photos and videos in a way that’s accessible and convenient but also cost-effective. Unlike a business that might store records on tape, Facebook can’t afford to let users wait that long to access something, nor can it afford to build data centers that keep photos in caches next to the servers (those Fusion-io machines aren’t cheap!).

“The current data center and hardware design is actually very sub-optimal for that problem,” Parikh says. “People say to us we should just use tape, but I’d rather poke my eyes out with chopsticks.”

So instead he’s thinking about a new storage-based data center as described in this article, with servers that turn on and off as needed and take up a lot of floor space but consume very little power. He added that the company is building software to handle moving data between different regimes of popularity and that changes the properties of the stored files as they migrate from one location to another. The product team shouldn’t have to think about that, said Parikh, the infrastructure should.

“Where we are going to be innovating at the data center, the hardware, the operating system and the kernel level,” he said. “All of that needs to be rethought. You can’t do this by shoving it into existing computing environments. You need a separate storage facility and other, bigger data centers, and different physical building and different design.”

Unlike today’s focus on getting the most performance from a watt of power, generally in a dense computing array, the cold-storage problem requires a lot of floor space and machines consuming the least power possible. So, in Prineville, Ore., next to its large web-serving data center, Facebook is essentially building out a 62,000-square-foot attic for your digital junk.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Locating data centers in an energy-constrained world
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

Opportunity 1: Democratize Hadoop

Ping Li

The opportunity, of course, lies in taking those ideas and turning them into products that the business world will eat up. Or, as Accel Partners’ Ping Li put it during a recent phone call, there has been a lot of innovation at the infrastructure layer with Hadoop and NoSQL, “but I’ve had this constant search for ‘now what?’” Some of these applications are popping up — they’re the types of things Accel is looking to invest in via its Big Data Fund — and Li thinks what’s going on in Facebook could serve as the inspiration for even more.

“Pretty much everyone at Facebook is talking to data that’s coming out of a Hadoop cluster,” he said, and they’re not all writing MapReduce jobs.

Fortunately, Facebook VP of Infrastructure Engineering Jay Parikh was on the same call, and he shared a glimpse into how Facebook is boiling big data down to bite-size pieces. Essentially, he said, Facebook uses Hadoop for just about everything, from friend recommendations to ad targeting to analyzing the efficiency of its data centers. But serving all of these uses means making sure the users in each department can actually interact with Hadoop in a meaningful manner.

Jay Parikh

Thanks to a collection of custom-built tools, user interfaces and visualization layers, he said, “we have a lot of [non-technical] users at Facebook that are able to run reports and view analytics [powered by Hadoop].” Already, a couple of former Facebook employees who helped invent Hive have launched Qubole, a cloud-based version of Hive that provides on-demand access to Hadoop with Hive’s signature SQL interface, but there’s a lot more that can be done.

And having easy access to Hadoop isn’t just about adding another tool to someone’s belt; ideally, it’s also about getting rid of a couple others. Doing big data right means doing it efficiently, Parikh said, so Facebook puts a lot of effort into designing tools that let users do a lot of things that previously might have required multiple products. They might be different than what users are used to, but hopefully they let users innovate even faster.

Opportunity 2: Look beyond Hadoop

Once you get outside the realm of established infrastructure tools such as Hadoop and NoSQL stores, though, things start to open up. “We have many things in the oven,” Parikh said, noting Facebook’s heavy use of MySQL, a graph database it has built and the new types of backends it had to build for Timeline and Newsfeed. “A lot of this boils down to the different needs of the project.”

Li agreed, noting the number of startups he sees that want to use Hadoop because it’s free and open source, but that end up having to do a lot of their own work to make Hadoop do what they want. Some, like Precog, which I recently profiled, opt to build their own products from scratch rather than try to fit them into Hadoop’s mold. “There are plenty of big data problems that have nothing to do with Hadoop, in my opinion,” Li said.

Still, there’s a balancing act that startups must perform when it comes to picking a platform on which to build applications. “The one thing I do caution entrepreneurs about is that good-enough is the enemy of great,” Li said, before adding that you won’t always be able to out-innovate community support for what’s already out there.

Opportunity 3: Go big. Like, data center big.

Anyone feeling particularly ambitious, however, might look at Facebook’s new deep-storage data center strategy and try to figure out a way to take that mainstream. The strategy, which emerged in August — and which I assume will come up when Parikh discusses Facebook’s infrastructure strategy at our Structure: Europe conference next month — involves designing data centers from the ground up to handle longer-term data storage for rarely accessed information instead of a steady stream of web transactions. The hardware, network and data center designs all need to be rethought for what Parikh calls the “changing temperature of data.”

The construction site for Facebook’s deep-storage data center.

“It’s not incremental [change],” he said, “I actually think it’s very different.” Energy-dense data centers that try to suck every last bit of power for computing will give way to ones that need far less power for processing, but still need to deliver data to users and analytic engines when it’s needed. Parikh calls it a huge challenge that could become very important to all businesses as more of them decide to keep data around for regulatory purposes, to serve users or just for rainy-day analytics projects.

The good news for startups: Facebook will open source some of its design work via the Open Compute Project, and some of the data-management work will manifest itself in the Apache Hadoop project. They just need to do the rest.

Feature image courtesy of Shutterstock user Anneka.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• 9 Companies that Pushed the Infrastructure Discussion in 2010

Sure, many of Europe’s cloud deployments may be private, but European companies are eager to learn what their counterparts in the U.S. are doing and share their own experiences. That’s why we’re mixing our Structure stars such as Amazon’s CTO Werner Vogels and Jay Parikh, VP of infrastructure engineering at Facebook, with Europeans such as Bernard Dallé, a partner with Index Ventures and Eirikur Hrafnsson, the founder of GreenQloud.

And we’re wrapping up the first day with a showcase of our five finalists. I’ve listed them below, but for more information you’ll have to come to Amsterdam on Oct. 16 and 17th and watch them present onstage. I can’t wait.

• Besol: This Spanish company has built a cloud management product that can span different clouds. The product, called Tapp, is one of those that could enable a real cloud brokerage model to develop.
• ComodIT: This company is building a software-as-a-service offering that will allow companies to create different deployment configurations for their servers and store them online for later access.
• DoxOut: This is a productivity app for the post-PC era, which in laymen’s terms means it will work on a variety of devices. The company says it does that without coding different apps for different platforms. It can work as an installed app offline or via a web browser when the user has connectivity.
• FileSpirit: Find corporate files in the cloud using the File Spirit app. The company is competing with Dropbox, SugarSync but says it offers file encryption and other services designed to appeal to corporate users.
• Wercker: In this devops focused world, companies can deploy dozens or thousands of code updates a day. Big companies write custom software to push out their updates, but Wercker has created a SaaS platform that will prep and test deployment software and integrates with GitHub.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• The fourth quarter of 2012 in cloud
• How direct-access solutions can speed up cloud adoption