Without question, enterprise data volumes have grown immensely, and organizations have indeed begun to recognize the value hidden in these larger stores. According to a 2011 study by the Aberdeen Group, organizations that effectively integrate complex data are able to use up to 50 percent larger data sets for business intelligence and analytics, to integrate external unstructured data into business processes twice as successfully, and to slash error incidences almost in half. The connection between a company’s success and its ability to leverage big data is very clear. In that sense, the media firestorm around big data has been completely valid.

Don’t believe the NoSQL hype

However, much of the buzz treats big data as though it remains to be conquered. The hype surrounding NoSQL, for instance, makes it seem like it’s the only database management system that can effectively manage big data and that, without it, immense value would remain untapped.

The first iterations of the NoSQL relation provided knee-jerk solutions for companies such as Amazon and eBay that needed to solve a crushing scaling problem – and fast. Although it solved the scaling issues, NoSQL isn’t the best solution for today’s complex enterprise-class applications. For one, NoSQL technologies all use their own proprietary coding interfaces. So moving to NoSQL creates headaches and a drain on resources, because database administrators and programmers have to learn new skill sets.

Additionally, in Hadoop, for example, links between data are not automated as they are in relational or object-oriented engines, and they must be manually joined by the developer writing some custom code to do the set operation. Plus, these newer technologies don’t yet play well with enterprise-class management and monitoring protocols, making them a high risk factor for mission-critical applications. Since these are only a few of NoSQL’s pitfalls, thinking that a solution to big data problems still doesn’t exist is in some ways warranted.

The reality, however, is quite the opposite.

Other technologies have been tackling big data for years

The zero-to-hero perception of big data neglects the fact that many companies and industries jumped on the big data bandwagon long ago. When the amount and complexity of the data became too much for relational database management systems (RDBMs) to handle, big data pioneers began using somewhat more obscure technologies such as object oriented systems and databases (ODB).

At this point, you might be thinking, “Okay, but can a lesser-known technology really tackle big data better than the latest cutting-edge innovation?”  Based on the following three examples, I’d say the answer is “yes.”

1. Big data on the rails

The U.S. Federal Railroad Administration, expecting rail freight traffic to double by 2020, created the RailEdge Movement Planner application to perform analysis of highly complex object models more than 30 times faster than an RDB. In shipping, time really is money: fuel consumption and delivery times are determined almost exclusively by the availability of an aging infrastructure. Yet there is money to be made by mastering minutiae and affecting change in real time rather than relying on predicted outcomes.

RailEdge organizes minute details and readings from a vast network of information sensors and physical items — the number of engines and cars per train, payloads, rail traffic, congestion at depots, etc. — all against the backdrop of time. That is worth repeating. Analyzing all this data against time creates really big data. RailEdge has improved average train velocity and fuel-efficiency and saved about $200 million in annual capital and expenses.

2. Big data in the air

Processing airline tickets poses an even bigger big data challenge. The massive amount of transactional throughput of Travelocity.com and other online ticketing services puts huge pressure on databases to handle every detail quickly and with perfect accuracy.

To solve this problem, Travelocity uses the SabreSonic Inventory System, the world’s most popular ticketing inventory solution. The big-data needs of the participating airlines — 30, at last count — requires an object-oriented system to maintain high performance and minimize IT infrastructure costs.

Harnessing big data to quickly and accurately process millions of transactions per day has saved Travelocity.com money and boosted the brand’s reputation. The system allowed them to switch from using multi-million-dollar mainframe hardware to relatively low-cost commodity infrastructure without sacrificing performance. Even more impressive? Reliability: Since turning the system on almost four years ago, it has never gone offline. Ever.

3. Big data on ice

My favorite story about the forgotten heroes of big data is one that delivered value beyond mere dollars and cents.

Tracking the effects of Arctic ice sheets on the world’s climate is an intricate process that requires scientists to monitor incredibly minute pieces of both historical and contemporary data in petabyte volumes. The National Snow and Ice Data Center’s (NSIDC) scientists needed to process billions of complex data objects to analyze how changes in Greenland’s ice sheet over time have affected global climate. To boot, the system also needed to enable real-time queries of the data to answer new questions about the ice sheet’s changes as the researchers made new inferences.

Traditional databases can’t do this. Hadoop can’t do this. In those systems, the relationships have to be rebuilt for every query. The only way for the NSIDC to do this was to drive an object-oriented model deep into the database’s architecture.

Without the system that the NSIDC developed, processing this amount of information would have taken years, rendering any results a matter of historical record rather than actionable intelligence.

Harnessing big data to create real value is certainly taking a quantum leap in necessity across every industry. But while today’s media hype surrounds predictive analytics and NoSQL technologies, we shouldn’t forget that tried and true technologies are out there that have been the silent heroes of big data for years.

Robert Greene, vice president of technology at Versant Corporation, has more than 15 years experience working on high-performance, mission-critical software systems. He provides the technical direction for Versant’s database technology, used by Fortune 1000 companies, including Dow Jones, Ericsson and China Telecom. 

Some rights reserved by Creative Tools.

For more on this big data phenomenon, be sure to check out GigaOM’s Structure:Data Conference in New York City on March 21 and 22.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• The importance of putting the U and I in visualization
• A near-term outlook for big data

Wednesday, GigaOM Pro published a report I wrote about the fast-growing ecosystem of Hadoop vendors, projects and users. The underlying theme of that report, and right now for big data in general, is software: the products, algorithms and languages that help companies process, analyze and act upon their mountains of information. In my experience, hardware rarely comes up in discussions about big data, and when it does, it’s generally centered on data warehouse appliances. But the omission hardly means hardware is irrelevant. In fact, big gear might become a big deal as companies look to bolster the performance of their big data systems.

Perhaps nowhere is this more clear than at the network level. As Sun Microsystems  Co-Founder Andy Bechtolsteim pointed out in a fireside chat at Structure: Big Data, the network has become the bottleneck in many systems as server hardware has continued to become faster and more powerful. This shouldn’t be a surprising sentiment, considering Bechtolsteim’s current company, Arista Networks, sells high-performance networking gear — including a new pair of low-latency switches just announced this week — but it’s also true. It’s why, for example, mobile-app-analytics startup Flurry recently upgraded its network infrastructure with Arista gear to handle skyrocketing network traffic, including across its growing Hadoop cluster.

When you’re moving terabytes of data across hundreds or thousands of nodes, or from one environment to another (e.g., from a Hadoop cluster to an analytic database) you can’t afford to have it lagging across the network for too long. If you’re looking to process and analyze in real time, latency needs to be as close to zero as possible.

Storage can be critical for better big data performance, too, as Cisco highlighted Wednesday with its new C260 M2 servers designed for OLTP and data warehouse applications. The new box can house up to a terabyte of memory and 16 solid-state or hard-disk drives (up to 9.6TB in total), which are important factors in its target use cases, as they must not only be able to store large amounts of data, but also be able to access it in a hurry to answer time-sensitive queries of feed applications. This is why large systems vendors such as Oracle, IBM and EMC sell massive appliances armed to the teeth with storage and memory capacity and running either transactional or analytic database software.

But Cisco’s server is somewhat unique in that it’s, well, a server, as opposed to a full-on appliance. It doesn’t seem like too big of a stretch to think some company that wants to seriously improve the performance of its Hadoop cluster would transition to the C260 M2 to get the performance gains of so much memory and SSD capacity while still being able to add to the cluster one rackmount server at a time. Further, a smaller cluster in terms of nodes means even less latency, because there are fewer network points for data to traverse as it crosses the system.

In their current states, many Hadoop clusters and other big data systems probably are just fine in terms of compute, network and storage performance. That’s why, as I explain in my report, and as I’ve reported here before, much work is underway to improve software performance for tools like Hadoop, while still focusing on less-expensive commodity gear. But that might change as companies start to rely more on analytics to make everyday business decisions.

Much like banks have spent untold millions on finely tuned software and hardware to carry out their risk analyses and high-frequency trading applications, more-mainstream companies also will start feeling the pressure to build analytic systems that deliver results in as close to real time as possible, even for batch processes. Certainly, Arista and Cisco aren’t alone in seeing this trend, and I suspect we’ll see a lot more network and server vendors targeting big data applications beyond OLTP analytic databases as the trend gains momentum.

Image courtesy of Flickr user jpctalbot.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Cloud computing infrastructure: 2012 and beyond
• The importance of putting the U and I in visualization

As I explained in detail last week, Hadoop has become popular among companies of all sizes, but most products at this point target broad use cases across industries. Yes, there’s still room for startups to get in here, but the door looks to be closing fast. It’s not just Hadoop, either; other techniques, from tradtional data warehouses to, arguably, predictive analytics, all are nearing the saturation point in terms of vendors selling the core technologies. Even a step up the stack from the core Hadoop layer are vendors like Datameer selling familiar-looking interfaces that abstract the complexities of processing and analyzing data with Hadoop.

But Ferguson made a particularly poignant, if not novel, observation: analyzing social media data is not the same, either in technique or in purpose, as analyzing user data to feed a recommendation engine for a site like Netflix. And herein lies the opportunity. Organizations keep on hearing about big data and about how big an opportunity it is, but even though the technology to capitalize on this opportunity is getting democratized, organizations still face a big challenge to hire personnel that understand not only the technology, but also how to ask right the right questions. Sure, analyzing social media data sounds great to find out what consumers like or how they might act sounds great, but actually being able to do it accurately is another issue. It’s a situation just begging for startups to fill the void between big data tools and actually using them for a particular task.

Whether the focus is by industry (e.g., tools for financial services, retail, etc.) or by use case (e.g., sentiment analysis, recommendation engines, etc.), one can easily envision an emerging class of companies tuning technologies like Hadoop or predictive analytics software to directly address these discrete classes of users. Organizations won’t necessarily need data scientists to “turn information into gold” if the data scientists employed by their software vendors have already done most of the work. Think about it like functions within spreadsheet applications tuned to specific industries, or like how PaaS startups took cloud computing a step further by configuring infrastructure with the push of a button. Just feed the application some data, push a button, and get results — no Ph. D. required.

To a degree, this is already starting to happen, but primarily by large vendors using their existing software (e.g., SAS for social media) and in the form of fairly limited-scope analytic technologies (e.g., graph databases), but I think these are just baby steps toward what could be a huge opportunity. Companies of all types want to be the next Yahoo or Facebook in terms of big data, and there are plenty of companies willing to help them do that in terms of infrastructure. The real opportunity now is in helping companies figure out how to use it.

Image courtesy of Pam Brophy.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

We kicked off the week with Om joining Nasdaq’s VP, David Wicks, to ring the closing bell. This is just the start of a closer partnership between GigaOM and the folks at Nasdaq.

That evening, we held a reception at the New York Times building, bringing together media executives, data scientists and journalists. A bunch of bloggers walked among the Times reporters’ Pulitzers — an interesting juxtaposition of old and new media.

All of this was a precursor to our main event: Structure Big Data, held at the Chelsea Piers. And it was a great success — not just in terms of attendance, though it was standing room only in the main hall. The sold-out event was also the first time that people from the Silicon Valley and New York tech communities, as well as leaders in finance, media and other New York-based industries, came together in one place to discuss how they were all using big data to gain valuable insights into their businesses. If you couldn’t make it, you can see some of the highlights in our photo gallery below or read through our coverage (with video clips) of the entire event.

Structure Big Data may have been our first big step into New York, but expect more to come. Last fall, we hired Ryan Kim, a crackerjack tech reporter from the San Francisco Chronicle, to be our “man in New York,” covering local startups and the growing app economy. In February, we hosted a Media Meetup, along with Automattic (see disclosure), to gather together the New York tech media, startup founders and venture capitalists.

This is just the beginning for us. It’s easy for tech publications to myopically focus on Silicon Valley, but increasingly, many of the exciting ideas and innovations are developing elsewhere. Our readers rely on us to find the next big ideas — wherever they are. That’s why more of our staff will be joining Ryan in New York, including Om himself who will return to his former life as a bicoastal writer. We’re looking forward to telling more of New York’s great tech stories. In the meantime, enjoy the photo gallery from Structure Big Data and a behind-the-scenes video from Nasdaq.

Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOm. Om Malik, founder of GigaOm, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How consumer media will change in 2013
• Social media in Q1: commerce and discovery dominated
• NewNet Q1: Content Farms and Niche Networks on the Rise

The Hadoop and Beyond Panel at Structure: Big Data

Hadoop, the open-source file system and MapReduce implementation for massive-scale data, was the talk of the conference Wednesday at our Structure Big Data conference in New York. From new Hadoop distributions to end-customers’ plans, Hadoop was all anyone could talk about. One of the companies whose name crept up in conversations was a stealth-mode company called Mapr, which is building a proprietary version of Hadoop and is likely to launch later this year.

Mapr, based in Saratoga San Jose, Calif., has been in the works for nearly two years. The Securities and Exchange Commission filings show the company has raised about $9 million in funding from Barry Eggers of Lightspeed Venture Partners and Peter Sonsini of the New Enterprise Associates.  On its web site, the company says it’s “engineering game changing Map/Reduce related technologies.” Its ambitions aren’t limited by that somewhat ambiguous statement.

People Behind Mapr:

• M.C. Srinivas Srivas, an ex-Googler  is the founder and CTO of the company.
• John Schroeder, formerly of Lightspeed VC and former CEO of Calista Technologies (acquired by Microsoft) and Rainfinity (acquired by EMC) is the CEO and co-founder of Mapr.
• The company has close to 30 employees, many of them based in India.
• Ted Dunning, chief scientist at Site Tuner and Veoh Networks, is the chief application architect at Mapr. He created the recommendation engine for Musicmatch, a music service that was popular before iTunes came on the scene. He is also one of the key guys behind the Apache Mahout data-mining project.

What Is Mapr Doing?

They are said to be building a proprietary replacement for the Hadoop Distributed File System that’s allegedly three times faster than the current open-source version. It comes with snapshots and no NameNode single point of failure (SPOF), and is supposed to be API-compatible with HDFS, so it can be a drop-in replacement.

The Road Ahead

Mapr might have an edge over Apache Hadoop in the interim, but Apache is working to improve the HDFS architecture in its distribution, and should have its own snapshot feature sometime in 2012. Also, Appistry sells a NameNode-free HDFS alternative based on its distributed CloudIQ Storage offering. As for the speed advantage, I don’t have any details for now, but if you have some thoughts, please share them with us.

On a broader canvas, I think Mapr is up against a whole lot of major competitors. Cloudera has a lead in the commercial market place, and the Apache Hadoop distribution on which it’s based keeps improving thanks to upgrades from contributors like Facebook and Yahoo. Apache Hadoop companies more control over their data, as they are not at all held hostage by a vendor, and surveys and anecdotal evidence alike suggest that Apache Hadoop is still the most widely-used version.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Big data 2013: key trends and companies to watch
• Real-­time query for Hadoop democratizes access to big data analytics
• Scaling Hadoop clusters: the role of cluster management

Aspera has built a proprietary high-speed file-transport technology, fasp, that helps data move across networks with issues like over-burdened WANs. Aspera is primarily the province of large companies dealing with big data, including digital media companies sending content among supply-chain partners, life sciences researchers sending genome-sequencing data among institutes and government intelligence customers sending video files between agencies.

Munson said current Internet infrastructure lacks three qualities:

1. availability
2. geographic independence
3. security

While all these issues need to be addressed in the fundamental architecture itself, the constraint has created an opportunity for Aspera’s transfer product. The reliability of Internet services is going up, which creates an expectation that this data will be available quickly, said Ammar Hanafi, general partner with Alloy Ventures.

While consumer web services can easily meet customer expectations, Aspera’s customers are a different story. “Our customers are moving many gigabytes and larger [quantities] of data that has to be chunked up and then distributed,” said Munson. But even if Aspera’s file transfer tech can make sure the delivery is as fast as the consumer web, the company has learned it can provide something else: predictability. “After solving the bottleneck, then you can offer customers predictability,” that manage their expectations, Munson said.

At the end of the day, its a physics problem, both Munson and Hanafi said. TCP, the transmission protocol used by IP networks, just doesn’t perform all that well for moving big data long distances. That’s both a big opportunity for startups like Aspera and big data infrastructure companies.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Will Facebook (or Apple) Be the Next Great Hadoop Champion?
• A near-term outlook for big data

Kahn Khan says the challenge lies mainly in three particular latency areas: data, analysis and decision making all take time from the initial trigger event to the final action event. And as more time passes at each of these hotspots, the value of any decision made from the data is reduced. As if that weren’t bad enough, I.T. shops are slowed by other challenges such as mobile computing adoption of employee-owned devices, the mobile commerce revolution and pressure to boost worker productivity.

What’s needed to respond to these challenges, according to Kahn Khan are new technology stacks, deployment models and shifts in both data and application paradigms. For starters, the cost of faster DRAM is now approaching that of high-end storage and should be considered for data. Using column stores of data can reduce the size of information arrays by several factors while the adoption of complex event processing engines (CEP) can bring real-time analytics. Data analysis also needs to be pushed down closer to where the data is actually stored — a tighter merging of application stores and data tiers, says Kahn Khan — and the use of in-memory analytics can also diminish information latency, leading to faster decision making in the enterprise.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Dissecting the data: 5 issues for our digital future

For example, Data Science Toolkit offers OCR functionality to convert PDFs or scanned image files to text files, filter geographic locations from news articles and other types of unstructured data or find political district and neighborhood information for any given location. Data Science Toolkit is available as a web service online, but it can also be downloaded and run on an Amazon EC2 or VM virtual machine.

Explaining the motivation for this release, Warden said during his talk that he has been living off of ramen noodles for years, but that didn’t stop him from getting creative with data analysis. “I’ve had my service living off this same kind of budget,” he said, adding: “You can hire a hundred servers from Amazon for $10 an hour.”

That’s exactly what Warden did last year after crawling Facebook and scraping 500 million pages that represented about 22 million users. He let Amazon’s servers loose on the scraped data, and 10 hours later had it boiled down to a database-ready format. “That was about a hundred bucks,” recollected Warden.

The result of these efforts was a massive analysis of friendship relationships on Facebook, which got him interviews with NPR, 500,000 visitors to his blog — and an angry call from Facebook’s chief legal counsel, who didn’t like what Warden was doing with the company’s data.

The conflict with Facebook lasted several months and ended up costing Warden $3000 in legal fees. “Big data? Cheap. Lawyers? Not so cheap,” he quipped. That episode may be one reason that Warden released the Data Science Toolkit under the GPL — no lawyers necessary to use it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Helix Nebula and the future of Europe’s cloud
• Platform as a Service in 2012
• Understanding and managing the cost of the cloud

Perhaps the most prominent example is Google’s geographic data used both in both the Google Maps and Earth products. The company harvests global information to create useful products in their own right, but each can be supplemented through localized user data. A modern data management web app makes it easy for Google to host, manage, allow collaboration and publication of data tables or personalized maps. For example, Google Maps data combined with information from hospitals and doctors can easily show which nearby health-care providers have flu vaccines available.

Making large amounts of data usable and modifiable by end users has the potential to create solutions that Google hasn’t envisioned yet. But what it has done is allowed for what Spector calls a “hybrid intelligence” because users and computers are doing more together than either could do individually. Scientists that track global warming may only have access to limited datasets which show only a small picture of the overall situation. Google Earth, however, can augment its base data with sensor information from various satellites and datapoints, providing a more holistic view of global warming.

This user community and data combination approach is leading to smarter machines as well. The voice search features offered by Google are becoming more accurate due to speech recognition data provided by users. In effect, the speech service is training itself because it’s learning from all of the incoming data.

Just as they can with Google Maps data, end users can leverage these smarter machines as well. Spector said that a spam-killing blog moderator could be created by end users if they train the system with both good blog posts and spam comments. Those inputs, combined with Google’s prediction APIs and Python scripts, would effectively create an intelligent automated moderator that could continuously improve its own performance.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012

But in order to use data to help deliver an impact for consumers, such as regionally customized clothing stores or a Starbucks that keeps serving hot caramel apple ciders through the summer months during colder years, the physical distribution chain has to change. Baum pointed out that grocery stores do a masterful job crunching data from their customers and about what is selling to them in order to predict inventory, but that inventory is still delivered on pallets in trucks and then loaded and unloaded by people.

He doesn’t see a huge wave of change coming at translating the data gains made in the digital world to the physical world anytime soon because many of those businesses aren’t appealing to venture capitalists, who are looking for less capital-intensive deals.

Much of his talk was about data interacting with people and enabling people to make faster and better decisions, but he also looked ahead to how machines can use data to interact and react in real time without human intervention. He gave the example of smart grids, which would involve appliances talking to the electric grid and utilities to determine the optimal time to use energy. For example, your dishwasher may wait till power is cheaper or the load on the electric grid is lighter before running. His thesis was that data can make the world more efficient, help companies make more money and improve the user experience, but we’ve got to solve the technology issues first.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Red-Hot Data Warehouse Market: Who’s Buying Next?
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data