GigaOM » big data analytics

Black box software: a problem for science that extends to big data

Amanda Alvarez — Thu, 16 May 2013 18:00:44 +0000

You probably don’t need to know how a calculator makes two plus two equal four, or how your favorite smartphone app works, but the way the background software is implemented can make a big difference to the output. Slight rounding errors or slow load times in these cases might be annoying, but when you scale up to big data modeling, for instance, you might want to take a closer look at the software running your calculations before you click go.

Blind trust in black box, or click-and-run, software is a growing problem in science, according to a commentary published Thursday in the journal Science, and the concern extends beyond formal research to other domains that use high performance computing.

The researchers who addressed the “troubling trend in scientific software use” were motivated by a growing unease that the abundance of powerful software is letting scientists derive answers without a thorough understanding of what the software is doing. Software snafus have been responsible for some high-profile data misinterpretations and retractions.

This wouldn’t normally cause a blip on the average citizen’s radar, but now a lot of these scientific conclusions have real-world implications, from climate modeling and weather forecasting to high volume financial trading. In any domain using big data, misplaced trust in the power of software can be problematic, particularly when the decision makers don’t know what the software they are using is doing, said lead author Lucas Joppa of Microsoft Research.

So what does ecology have to do with any of this? Joppa is an ecologist by training, and works on computational techniques in that field that may also have applications for big data more broadly. He and his colleagues surveyed scientists in a sub-field of ecology — species distribution modeling (SDM) — to find out how they choose software and how well they understand its inner workings.

“Lots of SDM techniques are only available as computational methods, but there is a lot of discourse going on in the literature about whether the methods themselves are correct,” said Joppa. Scientists use SDM to forecast where plants and animals will be in the future given current numbers, known habitats, and climate change. It’s a niche area of research, but the disquieting survey results should be noted in any domain where forecasting is done by plugging data into software.

Only 8 percent of the more than 400 scientists who responded had validated their modeling software against other methods. “The number speaks for itself,” said Joppa. “The real crux of the problem is the results from software being published in a peer-reviewed journal, versus the software itself having been peer-reviewed,” which is rare. Software packages, whether proprietary or not, are often black box systems that can’t be opened and inspected. Even if you can get under the proverbial hood, like with open source software, said Joppa, most people will still have no idea what they are looking at, or how to judge its quality.

To top it all off, having confidence in what your software is doing results in a massive computational catch-22: how do you know the software is giving you the right answer, if you can’t get the answer without running the software? The level of confusion over what algorithms are doing in the SDM field is illustrated by a debate over which of two statistical techniques is superior. It turns out, Joppa explained, that the two techniques were mathematically equivalent, but the ways they were implemented in software resulted in big predictive differences.

This sort of mix-up isn’t surprising given the messy nature of software development (if you can even call it that) in research environments. Joppa lauded efforts like Software Carpentry that teach scientists basic software fundamentals for better programming, and said the days of getting a doctorate by merely pushing a button are over.

“Scientists themselves can learn a bare minimum of software engineering,” said Joppa. On the flip side, he said computer science students should have more exposure to scientific methods. “People with traditional software engineering training become uncomfortable with the way scientists want to work with software, where the design and specs are constantly changing. The way that scientific software is built is fundamentally different from consumer apps.”

Developers of scientific software, like MathWorks or SAS, may want to watch this space. If Joppa’s suggestions are implemented, journals may start requiring that even proprietary software be opened up for inspection and peer-review. Nearly half of the surveyed ecologists report using free statistical language R as their primary software, so maybe there is hope yet, both for open, inspectable code, and for computational science becoming more accessible while yielding trustworthy, high impact results.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Sector RoadMap: Social customer service in 2013

Tue, 23 Apr 2013 06:55:38 +0000

“Social customer service” refers to those services that provide customer support via social media channels. Providing such services is no longer merely a niche or specialty sideline. Challengers, or disruptors who were early with the new technology, are working to expand and integrate their offerings into enterprise systems and processes.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

For big data achievements, IT and analysts need to work together

Jordan Novet — Wed, 20 Mar 2013 19:21:27 +0000

One trend emerging throughout GigaOM’s Structure:Data conference today is the collaboration between man and machine to solve big-data problems. Speaking with Phil Francisco, vice president of product management for big data at IBM, and Emile Werr, head of enterprise data architecture at the New York Stock Exchange, my colleague Barb Darrow spent a session Wednesday explaining how people — a company’s IT experts and business experts — sometimes need to work in different ways to achieve the same business goals.

Developers need to build systems for crossing lots of data sets from legacy data warehouses as well as Hadoop clusters and make available options for visualizing trends that might otherwise be obvious, Francisco said. That’s when business experts come into play and ask questions and derive insights that could lead to new strategies and campaigns.

How does that work in practice? Facing greater volumes of data, the NYSE has trained business analysts as “data architects” to develop a system with IBM products for capacity planning and spotting patterns to detect fraud in billions of transactions each day, Werr said. Analysts also need to be able to figure out if a a possible fraud case is a false positive. Those are early-stage use cases for analyzing data in near-real time.

For now, financial deployments tend to play out on premise. Werr pointed out places where public clouds make sense. Developers can test out new data architectures for data sets. But the cost advantage of running on production scale on Infrastructure as a Service (IaaS) such as Amazon Web Services is appealing, Werr said. But, at least for now, bandwidth across multiple data centers is an issue, he said.

Check out the rest of our Structure:Data 2013 live coverage here, and a video embed of the session follows below.

A transcription of the video follows on the next page

Go to page 2 (of 2) on GigaOM .

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

For big data analytics, recall the tried and true old-school rules

Jordan Novet — Fri, 08 Mar 2013 18:27:21 +0000

Data analysis didn’t start with Hadoop. Companies have been working with data to get insights for decades. While technology has changed, some of the rules from the past still apply, or ought to, as data gets bigger and bigger.

Jack Rivkin, an occasional blogger with deep investment experience, recently shared some of the best practices he was exposed to early in his career working on economic forecasts. He shared some sage suggestions for enterprises to bear in mind as they consider and implement big data strategies. Among his insights:

Forecasting models can only be as good as the data inputs.
Be skeptical and hedge when sharing the models by noting factors that could lead to different results.
The less time it takes to process data, the more valuable it is.
Constantly improve models and inputs.

Of course, big data isn’t wholly evolutionary — it does bring its own all-new opporunities and risks. Some of the world’s leading data scientists, IT executives and business users will address them at GigaOM’s Structure:Data conference in New York on March 20-21.

Feature image courtesy of Flickr user luckey_sun.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The Guardian’s data journalism is cool, but it can take three weeks to make

Jordan Novet — Wed, 27 Feb 2013 01:17:05 +0000

This headline and body of this story were corrected at 11:10 p.m. with a more accurate description for the typical period of time for the deployment of Guardian journalist Feilding Cage’s data visualizations. Also, Guardian Datablog Editor Simon Rogers was incorrectly described as Cage’s boss, and that reference has been removed.

Once he finds a suitable topic, Feilding Cage, a New York-based developer and journalist for The Guardian, can easily spend three weeks generating the source information and designing a visualization for what’s become known as data journalism. The results bring understanding and reader engagement to topics that are otherwise discussed with a lot of words or static numbers. Readers can and do play around with the information, share it widely and discuss it for long periods after it appears online.

The Guardian’s interactive guide to gay rights in the United States

Cage is one of a handful of Guardian journalists who generate reports that say new things about topics that pop up in the news or are just plain old interesting. Cage and Simon Rogers, editor of The Guardian Datablog and Data Store, spoke about their work at the Strata conference at Santa Clara, Calif., on Tuesday.

Along with The Guardian, a few other news organizations have been putting an emphasis on data-driven reporting and visualizations, apps and even games in the past few years, such as the Chicago Tribune, the Los Angeles Times and ProPublica (Check out the Data Journalism Handbook for more information on this sort of work.)

Data journalism and visualization stand out for the verification and occasional gray-area explanations that journalists provide. Cage, for example, accompanied his interactive visualization of gay rights in the United States with a blog post explaining his methodology and disclosing his assumptions.

Screenshot from the Zoomdata’s big data analytics iPad app

It’s certainly one way to say something fresh with data, but it’s time-consuming when you consider big data analytics apps that provide users with real-time information users can compare against Hadoop-processed historical data, such as Zoomdata. (That company, which my colleague Derrick Harris covered last year, released the beta version of its iPad app on Tuesday.)

It would be neat to find a happy medium for enterprises that want original insights that every employee can see and use and act on but doesn’t take three weeks to generate. That’s especially true because the return on investment for work like Cage’s is hard to identify, although it’s possible the content could indirectly generate revenue by driving users to content they have to pay for.

Bridging the gap might be a matter of finding the perfect data scientist for the company. Or it might be a matter of time before the kind of work Cage does is automated. A computer already can write an earnings story, although it might be a few years before computers put wordsmiths out of business.

Maybe it just doesn’t make sense to cross data journalism visualizations with big data analytics apps. But I, for one, would like to play with such a tool.

Entrepreneurs from companies that work with and make visualizations from big data, such as Quid, will speak at the GigaOM Structure:Data conference on March 20-21 in New York.

Disclosure: The Guardian is an investor in Giga Omni Media, which publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

DataDirect Networks brings out Hadoop appliance for enterprises

Jordan Novet — Tue, 26 Feb 2013 08:01:15 +0000

DataDirect Networks, a hardware vendor with roots in providing storage for high-performance computing, is introducing a Hadoop appliance for enterprises, adding another notch to the trend of going with hardware for big data deployments.

DataDirect built hScaler to meet the speed and performance needs of those customers while offering ease of use for enterprise customers keen on Hadoop. Speed aside, hScaler stands out because it does away with direct-attached storage and incorporates RAID architecture instead. It lets users scale computing and storage resources independent of one another, precluding the chore of swapping out a server when a disk fails, as my colleague Derrick Harris has written.

The hScaler appliance, which runs with the Hortonworks Data Platform, can move fast with InfiniBand storage capable of operating at 40 gigabytes per second. In a sample configuration, 504 terabytes of storage are possible in a rack. The rack is four times as dense as a conventional data center rack, requiring less spending for cooling and square footage.

Because they aim to speed up and simplify Hadoop deployments, appliances such as hScaler are catching on, and DataDirect Chief Technology Officer Jean-Luc Chatelain expects the trend to continue. Greenplum, Oracle, Teradata and other companies sell appliances capable of running Hadoop jobs. For the sake of taking advantage of easy and quick data analytics processing, Chatelain sees the Hadoop hardware trend only getting bigger.

Appliances could be useful for enterprises looking to run Hadoop jobs, as employees can save time and focus more on building applications. Big data veterans will talk about innovative uses of Hadoop and other big data technologies at the GigaOM Structure:Data conference on March 20-21 in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

VCs pour money into data startups during 2012

Jordan Novet — Fri, 22 Feb 2013 20:47:42 +0000

Venture capitalists made more big data investments in 2012 than in any previous year, according to a report released Thursday from CB Insights and law firm Orrick.

“As evidenced by financing and deal activity, Big Data is gaining steam,” the report stated. The number of big data venture deals in 2012 rose by nearly 20 percent as compared with the previous year, going from 132 deals to 164. In the fourth quarter of 2012 alone, there were 49 venture deals for big data plays.

Among them are Basho, ClearStory Data, Continuuity, Drawn to Scale, Mortar Data and WibiData.

Despite the bump in number of deals, the total amount of money VCs threw at big data startups in 2012 — $1.39 billion — was down by nearly 7 percent year over year. The median deal size decreased slightly, from $6 million to $5.7 million.

The recipients of the biggest big data investments of 2012 were, in descending order, Cloudera ($65 million), Palantir Technologies ($56.1 million), Rocket Fuel ($50 milion), 10gen ($42 million) and Nimble Storage ($40.7 million).

Which VC firms closed the most big data deals last year? SV Angel, which invested in 14 companies. Sequoia Capital and IA Ventures tied for third, with 13 deals each, followed by New Enterprise Associates with 12 and First Round Capital with 10.

The percentage of deals for big data infrastructure in relation to all big data funding continued to fall, while big data analytics has risen to a high of 48 percent of all deals. (The other category listed in the report is big data applications.)

By far, California has been the leading state for big data venture deals in the past five years, with 230 in 2008-2012. New York, with 67 in that same time window, and Massachusetts, with 57, lag far behind.

Assuming the trend lines continue, then, you would be most likely to get funded if you run a big data analytics company in California.

As for 2013, the report cites a few big data investments VCs have already made so far: Sailthru and Nomi.

Others include Ayasdi and Think Big Analytics, whose founder and CEO, Ron Bodkin, will speak at GigaOM’s Structure:Data conference in New York in a few weeks.

In addition to Bodkin, entrepreneurs from other venture-backed big data startups who will speak at the conference include Justin Sheehy, chief technology officer at Basho; Jonathan Gray, Continuuity’s founder; Bradford Stephens, Drawn to Scale’s CEO; and Doug Daniels, chief technology officer of Mortar Data.

Feature image courtesy of Shutterstock user extradeda.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Open-source search tool Elasticsearch gets $24 million

Jordan Novet — Tue, 19 Feb 2013 13:00:37 +0000

Open-source search provider Elasticsearch has secured $24 million in Series B venture funding, showing business demand for free and simple big-data analytics. Mike Volpi of Index Ventures led the funding round, which included contributions from Benchmark Capital and SV Angel.

Amsterdam-based Elasticsearch, which has now raised a total of $34 million, generates revenue by teaching people how to use the tool at training courses and help them solve problems by way of support subscriptions. Introduced in 2010 after founder Shay Banon developed it in his free time, the open-source Elasticsearch program today gets downloaded 200,000 times a month. Banon launched the company itself six months ago, when CEO Steven Schuurman got involved, in time to take on a Series A round.

Elasticsearch can make quick work of searching billions of documents and petabytes of data, structured and unstructured alike, said Banon, now the company’s chief technology officer. A single developer can use it to find needles amid haystacks of tweets and other kinds of data, eliminating the need for a team of data scientists, Banon said.

Like LucidWorks, Elasticsearch was developed on top of open-source Apache Lucene. But LucidWorks (which until August 2012 was named Lucid Imagination and has raised at least $16 million) focuses more on the enterprise, as my colleague Barb Darrow has reported, while Elasticsearch has caught on with startups and enterprises alike in several industries, according to Schuurman.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Bina launches box to analyze genomes; cloud on the way

Jordan Novet — Tue, 19 Feb 2013 05:01:15 +0000

Bina Technologies is launching its Bina Box for on-premise genome processing, enabling researchers to quickly and cheaply analyze genomes and give doctors data-driven suggestions for custom treatments.

Use a genome sequencer to see one person’s DNA profile, and you’ll get 6 billion unique characters, or half a terabyte of data, said Bina co-founder and CEO Narges Bani Asadi. Start processing it to find mutations and variations, and you’ll find yourself with more than one terabyte. It’s not small data. As the price of sequencing a genome keeps dropping, scientists will want to do this more and more. It’s a big data problem, Bani Asadi said. The company wants to solve the problem on premises, with hardware and software.

The Bina Box will run on “high-end Intel processors and very high-bandwidth memory,” Bani Asadi said, and can scale out with additional Bina Boxes as customers processing needs change. Price depends on how much processing customers have in mind. If a customer wants to process 100 samples a month, for instance, it would cost $12,500 per month, or $125 per sample, said Mark Sutherland, Bina’s senior vice president of business development.

A Bina Cloud to tie in with the Bina Box will come later this year. The Bina Cloud will host just the needle of genomic data isolated from among the haystack of the entire genome, and it will enable scientists to aggregate many genomes, run data visualizations and collaborate to derive big-picture insights. Early customers are already using a pilot version of the cloud.

The box offering contributes more proof of the notion that, for certain uses, public clouds might not make sense, not yet anyway. (It remains a largely popular perspective in financial services, as my colleague Barb Darrow reported a couple of months ago.) The Bina Box, for its part, “provides security that on-premise solutions have, versus cloud solutions, (which) sometimes people in this industry are not completely ready to move into,” Bani Asadi said. Big pharmaceutical companies are a perfect example, as a breach could hamper product development using genomes. Aside from security, there’s the matter of performance. “It’s impossible to send (half a terabyte of raw data from a sequencer) to the cloud easily,” Bani Asadi said.

Meanwhile, other genomics-focused startups, including DNAnexus and Appistry, are eschewing hardware and relying exclusively on cloud resources.

Whether hardware is involved or not, as my colleague Derrick Harris mentioned when he wrote about Bina last year, it’s clear that the rise of big genomics inherently equates to a rise in data.

The practice of merging life sciences and other industries with big data will come up in conversation when Ayasdi CEO Gurjeet Singh hits the stage at GigaOM’s Structure:Data conference on March 20 in New York.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

With new pricing scheme, ParAccel lets users analyze unlimited big data

Jordan Novet — Tue, 12 Feb 2013 14:00:01 +0000

Analytic database company ParAccel is addressing the realities of big data by letting customers pay a flat rate for storing as much data as they want, instead of paying by the terabyte.

The new licensing model, officially called Right to Deploy, supports what the company calls “unconstrained analytics.” The idea is that data scientists can stop worrying about how many nodes or terabytes they’re storing in ParAccel and instead focus on drawing insights from their data, said John Santaferraro, ParAccel’s vice president of solutions and product marketing.

Santaferraro declined to describe the formula for determining the price for a company to go with Right to Deploy, saying only that it will depend on the company and what the company wants to do with its data. Still, he believes it’s a good choice for clients. “It’s even more cost-effective than the per-node pricing (model),” Santaferraro said. “There’s no reason why customers wouldn’t want to do it.”

Right to Deploy seems to make sense as more companies are embracing big data and storing more data than ever before in order to run more thorough and complex analytics. GigaOM Research analyst Lynn Langit expects other companies to try the model “as (clients) want to keep more of their data,” she wrote in an email.

Amazonbecame a ParAccel investor in 2011 and gained more exposure in November when Amazon Web Services announced its Redshift data warehouse service, which incorporates technology licensed from ParAccel.

Big data luminaries will discuss use cases, challenges and achievements at the GigaOM Structure:Data conference in New York on March 20-21.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.