Blind trust in black box, or click-and-run, software is a growing problem in science, according to a commentary published Thursday in the journal Science, and the concern extends beyond formal research to other domains that use high performance computing.

The researchers who addressed the “troubling trend in scientific software use” were motivated by a growing unease that the abundance of powerful software is letting scientists derive answers without a thorough understanding of what the software is doing. Software snafus have been responsible for some high-profile data misinterpretations and retractions.

This wouldn’t normally cause a blip on the average citizen’s radar, but now a lot of these scientific conclusions have real-world implications, from climate modeling and weather forecasting to high volume financial trading. In any domain using big data, misplaced trust in the power of software can be problematic, particularly when the decision makers don’t know what the software they are using is doing, said lead author Lucas Joppa of Microsoft Research.

So what does ecology have to do with any of this? Joppa is an ecologist by training, and works on computational techniques in that field that may also have applications for big data more broadly. He and his colleagues surveyed scientists in a sub-field of ecology — species distribution modeling (SDM) — to find out how they choose software and how well they understand its inner workings.

“Lots of SDM techniques are only available as computational methods, but there is a lot of discourse going on in the literature about whether the methods themselves are correct,” said Joppa. Scientists use SDM to forecast where plants and animals will be in the future given current numbers, known habitats, and climate change. It’s a niche area of research, but the disquieting survey results should be noted in any domain where forecasting is done by plugging data into software.

Only 8 percent of the more than 400 scientists who responded had validated their modeling software against other methods. “The number speaks for itself,” said Joppa. “The real crux of the problem is the results from software being published in a peer-reviewed journal, versus the software itself having been peer-reviewed,” which is rare. Software packages, whether proprietary or not, are often black box systems that can’t be opened and inspected. Even if you can get under the proverbial hood, like with open source software, said Joppa, most people will still have no idea what they are looking at, or how to judge its quality.

To top it all off, having confidence in what your software is doing results in a massive computational catch-22: how do you know the software is giving you the right answer, if you can’t get the answer without running the software? The level of confusion over what algorithms are doing in the SDM field is illustrated by a debate over which of two statistical techniques is superior. It turns out, Joppa explained, that the two techniques were mathematically equivalent, but the ways they were implemented in software resulted in big predictive differences.

This sort of mix-up isn’t surprising given the messy nature of software development (if you can even call it that) in research environments. Joppa lauded efforts like Software Carpentry that teach scientists basic software fundamentals for better programming, and said the days of getting a doctorate by merely pushing a button are over.

“Scientists themselves can learn a bare minimum of software engineering,” said Joppa. On the flip side, he said computer science students should have more exposure to scientific methods. “People with traditional software engineering training become uncomfortable with the way scientists want to work with software, where the design and specs are constantly changing. The way that scientific software is built is fundamentally different from consumer apps.”

Developers of scientific software, like MathWorks or SAS, may want to watch this space. If Joppa’s suggestions are implemented, journals may start requiring that even proprietary software be opened up for inspection and peer-review. Nearly half of the surveyed ecologists report using free statistical language R as their primary software, so maybe there is hope yet, both for open, inspectable code, and for computational science becoming more accessible while yielding trustworthy, high impact results.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Health care and big data in 2012
• The importance of putting the U and I in visualization
• Pervasive Software retools for cloud, big data: will it be heard?

The Periodic Table of iPhone: Rare earth metals and how they are used inside smartphones

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The state of cross-platform media measurement
• How emerging technologies will influence collaboration
• Analyzing the wearable computing market

A new study has found that people who were shown a Facebook status update and then a sentence from a book were 1.5 times more likely to remember the status update. And users who were shown a status update and a photo of a person’s face were 2.5 times more likely to remember the text than the photo. While much of the focus in social media is centered on photos right now, the research validates the idea that status updates can have real staying power with our brains, because we recognize it as something closer to actual human speech.

The study, conducted by researchers from the UK and U.S. and published in the Memory and Cognition journal, found that text written for Facebook updates fell into the category of “mind-ready” information, meaning our brains more easily took in the text and committed it to memory because of its conversational, unedited nature that’s more like human speech. And in a second test, they found that it wasn’t just the gossipy nature of Facebook that had an impact — conversational tweets or online comments had a similar effect. (So maybe all those grammar-challenged updates from your friends aren’t as awful as you thought, as long as they sound like something your friends would say.)

In other words, you’re more likely to remember this:

“I updated my grilling app, iGrill, today and it now has Facebook integration that lets you see what other people are grilling right now around the world. Awesome. I’m making a Fred’s steak.”

Than this:

But aside from giving you a pass on your Facebook grammar, the study has implications for almost everyone using the platform. The researchers noted that Facebook is updated more than 30 million times an hour. As the company works to build out its search capabilities and give you access to all the text and information posted from your friends, it’s worth noting that the text — and not just the photos that Facebook is known for — has real value. And that advertisers and marketers posting content on the platform should think closely about what resonates.

Aka, we should totes wr8te 4 actual people, bro.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Social 2013: The enterprise strikes back
• Examining the rise of crowd labor platforms in 2012
• Social third-quarter 2012: analysis and outlook

In July, three UK education research councils and the European Commission announced stipulations that future research partly funded by taxpayers – much of which is currently published through subscription journals – must be made more open-access. The UK government has labelled research “paywalls” “deeply unhealthy”, and wants to free up availability.

Berstein Research‘s Claudio Aspesi writes in a research note:

“It could drive the profitability of the journal business of Elsevier down by as much as 60%…

“Elsevier journal revenues would be under significant threat because the article processing charges it would earn for many of its publications are unlikely to prove anywhere near what the company needs to be revenue neutral…

“We think the risk posed to the Elsevier business model is substantial. We believe investors are underestimating the disruption that both the EC and even the UK policies could pose to the business model of Elsevier…

“A collapse of the profitability of Elsevier would be catastrophic for Reed Elsevier.”

Reed Elsevier’s share price has gone on rising through the recent announcements due to healthy recent results…

Open-access models may be largely confined to the UK unless Europe forces member states to adopt similar policies. But, with the U.S. also planning similar moves, the publishers will need to adapt.

In May, science minister David Willets told journal publishers gathered at a Publishers Association conference:

“I realise this move to open access presents a challenge and opportunity for your industry, as you have historically received funding by charging for access to a publication,” Willetts told publishers.

“Nevertheless, that funding model is surely going to have to change … To try to preserve the old model is the wrong battle to fight. Look at how the music industry lost out by trying to criminalise a generation of young people for file sharing.”

Many researchers were already revolting against health and science journal publisher Reed Elsevier for selling bundles of journals containing their work, rather than individual journals, to libraries. Tens of thousands of people signed a petition.

Alongside Elsevier in the Reed Elsevier stable is Variety publisher RBI. Bernstein thinks Reed Elesvier, which aborted a planned RBI disposal during the worst of the downturn, should break itself up but that it is more likely to retain an RBI that continues to down-size itself.

Disclosure: Reed Elsevier is an investor in paidContent parent GigaOM through its Reed Ventures arm.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s tactical retreat on privacy
• Google and the Ghost of Silicon Valley Past
• NewNet Market Overview, Q1 2010

Co-founder and CEO Victor Henning told me that the company’s API, which offers other services access to its trove of millions of scientific documents, has just surpassed 100 million calls each month. A year after launching the service — which provides access to the information stored in around 65 million scientific papers, documents and files in Mendeley’s databases — the site has around 240 apps that employ it, and is now seeing growth rocket. And that growth, he said, comes through the increasing popularity of those apps, not through any new, specific effort on its own part.

Henning highlighted a few of the projects which are creating the bulk of that traffic, including some interesting examples of the benefit opening up some data can have on what is, traditionally, quite a closed market.

For example, Readermeter.org and Total Impact are both services that measure how much impact a particular scientific paper or author has by analyzing how much and how widely the work is read. Meanwhile productivity app Hojoki integrates with Mendeley, pulling updates in alongside other services to create a personalized newsfeed for you.

Perhaps most interesting of all is OpenSNP, a project that allows people to share their genomic data with each other. It’s using Mendeley to help users cross reference the data they are finding out about their own genetic makeup with the latest scientific research as a way of understanding what’s going on in their bodies. If that’s not mind-blowing, I don’t know what is.

Henning says this is all part of a move to opening up science.

Next up? Wary of Twitter’s recent API troubles, Henning says that Mendeley wants to enable as many third party apps as possible, and has no plans to force money out of them — instead focusing on paid accounts and secondary services like its institutional dashboard.

“In academia, everything revolves around journals — but everything is behind a paywall, only available to universities who pay expensive subscriptions, and without APIs or other ways to access the data,” he said. “People see the importance of this [Mendeley's API] for opening up science.”

“We want to keep growing the ecosystem, get apps talking to each other and get more integration with the Mendeley experience” he adds. “But we don’t intend to monetize the apps.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s tactical retreat on privacy
• The 2013 task management tools market
• How consumer media will change in 2013

The answer: scientific research.

More specifically, large amounts of scientific work is funded by government agencies — yet the results end up hidden behind paywalls. Money from the public purse is used to pay academics to undertake investigation and write up the results, before academic journals take over copyright and sell access to the work for profit. Citizens are effectively being asked to pay twice for any information: first to fund the research, then to access it.

Over the last few years, a growing movement has been pointing out this illogical situation and campaigning for what’s known as “open access”: unfettered availability to research that’s funded by the public purse. And on Monday that movement appears to have won a significant victory, with the British government announcing that it would make open access a condition of any public funding in future.

Science minister David Willetts said in an announcement that the new system would be implemented by 2014, meaning that any research that uses public money could no longer be locked away.

“Removing paywalls that surround taxpayer funded research will have real economic and social benefits. It will allow academics and businesses to develop and commercialise their research more easily and herald a new era of academic discovery. This development will provide exciting new opportunities and keep the UK at the forefront of global research to drive innovation and growth.”

This may be one of the most significant victories for the open movement so far — and advocates will certainly be hoping that other governments, including the U.S., follow suit sooner rather than later.

It’s all part of a sea change happening in science towards openness, a move that’s taking place in large part because of the internet.

Just as the net has disrupted other industries, so it’s starting to make science more collaborative, more accessible and more democratic. My colleague Mathew Ingram has documented much of this conversation, and I’ve written about it before too, talking to startups like ResearchGate, which is trying to turn the stuffy world of research upside down by helping scientists collaborate.

And last year I covered the story of open science, talking about the work of Timothy Gowers, a mathematician at the University of Cambridge who has been one of the lightning rods for openness.

There are many interpretations of what open science means, with different motivations across different disciplines. Some are driven by the backlash against corporate-funded science, with its profit-driven research agenda. Others are internet radicals who take the “information wants to be free” slogan literally. Others want to make important discoveries more likely to happen. But for all their differences, the ambition remains roughly the same: to try and revolutionise the way research is performed by unlocking it and making it more public.

Gowers later led a boycott against the publisher Elsevier which drew in some 12,000 academics, who all objected to the company’s approach.

So, after all that, a victory — if a relatively small one.

Before anyone gets too excited, however, it’s also worth noting that this change is not without pain. As The Guardian notes, this process itself doesn’t come free — and may eat into already-strapped science budgets.

Though many academics will welcome the announcement, some scientists contacted by the Guardian were dismayed that the cost of the transition, which could reach £50m a year, must be covered by the existing science budget and that no new money would be found to fund the process. That could lead to less research and fewer valuable papers being published.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in GigaOmniMedia, the company that publishes GigaOM.

Scientist photograph copyright Shutterstock / damicoangie

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Google and the Ghost of Silicon Valley Past
• Will cloud computing push the BRIC market to the front?
• Facebook’s tactical retreat on privacy

Scientists aren’t worried about storing or processing all data, according to an article written by Mari Silbey for A Smarter PlanetSmartPlanet. Instead they’re worried about shipping that data from Chile to everyplace else it will be wanted. Basically it’s not a big data issue, it’s a broadband issue.

The proposed telescope would take two photos of the night sky every 60 seconds and then process all of those 6 gigabyte images during the 12 hours of daylight it isn’t snapping pics. The project plans to release annual data analysis reports which include the highest resolution photos for the broader scientific community, but that data will run into multiple petabytes of information each year. From the post:

The new synoptic telescope is being built on Cerro Pachon, a mountain in Chile, but the images it collects will be transported nightly to North America and up to the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. That will require a huge amount of bandwidth, and bandwidth is increasingly a scarce resource in our data-hungry world.

Moore’s law observes that computing power doubles every one and a half to two years. However, bandwidth isn’t on course to increase at nearly the same pace. And that, says Borne, is a problem: “The number one big data challenge in my mind is the bandwidth problem; it’s just having a pipe big enough to move data fast enough to whatever end-user application there is.

The solution is both fatter and faster pipes over which to transmit such data. The post lists solutions like Globus Online and the U.S. government’s Project Ignite that will make building new networks easier, but there are also some technological solutions such as the special file transfer protocol used by Aspera that could help. But other than fatter or faster pipes, is a solution that is perhaps more promising both for science and for business use — better algorithms to reduce the data sent. From the post:

The way algorithms help minimize the bandwidth issue of distributed data is by reducing the amount of information that needs to be transported for study. For instance, instead of transferring an entire set of raw data, scientists can employ relatively simple algorithms to reduce data to a more manageable size. Algorithms can separate signal from noise, eliminate duplicate data, index information, and catalog where change occurs. Any of these data subsets are inherently smaller and therefore easier to transport than the raw data from which they emerge.

So maybe it’s a method of deduplicating data before it travels over the congested networks or maybe it’s a form of pre-processing on site that can reduce the unnecessary data or at least group data into different cohorts so scientists can pick only the data sets that are relevant to their research.

This astronomy project isn’t the only one dealing with an influx of data. I’ve covered the proposed Square Kilometre Array radio antenna project that aims to look all the way back to the creation of the universe as another example of where we’re going to have to rethink how we handle data — although the SKA folks are worried about the processing and storage of what they estimate will be about 300 to 1,500 Petabytes of data each year.

But as researchers encounter these roadblocks — or bandwidth bottlenecks — the solutions they and the computer science industry come up with will also help enterprises filter through their own big data. For example, algorithms that are designed to detect outliers as a way to clean up irrelevant data might be tweaked to help detect fraud at financial institutions, according to the post. Really, anything that helps people parse and analyze their huge stores of data will be welcome, especially given how much hope everyone from government watchdogs to marketers have in the promise of big data.

Images courtesy of the LSST project.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• How direct-access solutions can speed up cloud adoption
• The importance of putting the U and I in visualization

The news will prove unpopular with academic publishers, which license and peer-review researchers’ work and charge libraries to make it available.

“As taxpayers put their money towards intellectual enquiry, they cannot be barred from then accessing it,” science minister David Willetts said in a speech to the Publishers Association on Wednesday (transcript).

“They should not be kept outside with their noses pressed to the window – whilst, inside, the academic community produces research in an exclusive space.”

A group led by Dame Janet Finch will shortly advise the government on how to accomplish Willett’s aims online. But Willetts revealed it is likely to moot a “green” option, which would see journal publishers granted a short exclusive window on publishing publicly-funded research, and a “gold” option, under which the research would be openly available from the start.

“I realise this move to open access presents a challenge and opportunity for your industry, as you have historically received funding by charging for access to a publication,” Willetts told publishers.

“Nevertheless, that funding model is surely going to have to change … To try to preserve the old model is the wrong battle to fight. Look at how the music industry lost out by trying to criminalise a generation of young people for file sharing.”

Many researchers were already revolting against health and science journal publisher Reed Elsevier for selling bundles of journals containing their work, rather than individual journals, to libraries. Over 11,000 people have signed a petition.

Wresting exclusivity away from journal publishers could destroy some of their business value.

The UK is currently creating a portal, Gateway To Research, to provide links to published publicly-funded research and some of the data sets which underpin them. Jimmy Wales is advising on format standards.

Willetts acknowledged journals provide an important peer-review role but revealed himself to be an apparent opponent of publishers charging for content online:

“Perhaps I might speak from the experience of writing my own book, The Pinch, on fairness between the generations.

“It was very frustrating to track down an article and then find it hidden behind a pay wall. That meant it was freely accessible to a professional in an academic institution, but not to me as an independent writer.

“That creates a barrier between the academic community and the rest of us, which is deeply unhealthy.”

The UK government is aligned with the European Commission, which has previously said it wants to see more free access to publicly-funded research and more open data, and claims the US Committee on Economic Development is moving in the same direction.

International consensus on the moves would be important else UK researchers could find themselves giving away their research to the world online whilst having to pay to access research from other countries, Willetts said.

Disclosure: Reed Elsevier Ventures is an investor in GigaOM, the publisher of paidContent.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Google and the Ghost of Silicon Valley Past
• Social media in Q1: commerce and discovery dominated
• Controversy, courtrooms and the cloud in Q1

Domain expertise, statistics and HPC, unite!

The latest is Bina Technologies, which just emerged from stealth mode last week and is bringing an Apple-like business model to genomics. The company, which grew out of a research project at Stanford University, relies on its Bina Box appliance to make genome analysis faster than typically possible without the benefit of having a supercomputer and a research network on hand.

According to Bina CEO Narges Bani Asadi, who co-founded the company while completing her Ph.D. at Stanford, the appliance came about as part of a mission to solve a disconnect among the stakeholders in cancer research. Improving the analysis of cancer data required input from medical researchers, statisticians and high-performance computing experts, “but people are not speaking even the same language,” she said. While they’re all headed in the same direction, their paths rarely converge to harness peak velocity.

Bani Asadi and her team solved that problem by developing a system that merged the three areas into one. With Bina, researchers can develop analysis pipelines that are optimized at both the algorithmic and silicon levels to run optimally across a mix of CPUs, GPUs and FPGAs, all of which are present within the purpose-built box. Applications are getting what they need in order to perform their best, and Bina says results can be processed 10 to 100 times faster (hours instead of days) than running jobs on the Amazon Web Services cloud, which has proven very popular for genomics workloads thanks to its supercomputer-like performance.

That being said, a chart Bina uses to illustrate the performance difference compares the Bina Box to a single eight-core AWS instance rather than a cluster of those high-performance instances.

The Apple analogy

Bani Asadi answers the inevitable question of whether research centers will want to special appliances instead of using the cloud or generic servers by pointing to Apple. That company’s devices and computers can be a little more expensive and more difficult to tinker with than alternatives, but they’re also designed specifically with Apple’s operating system and applications in mind. It’s an analogy other companies, such as cloud computing startup Nebula, also use to justify their appliance-based businesses.

Not that Bina dismisses the cloud. If Bina’s software is the Mac OS to the Bina Box’s iMac, the Bina Cloud is the company’s iCloud. Once the box processes the raw sequencing data and compresses it into a smaller volume (up to 1,000 times smaller), the data is shipped to the Bina Cloud where it’s stored and can be easily accessed and shared. Actually, Bani Asadi said that’s where the most-innovative research likely will take place. While Bina’s appliance handles the necessary first steps of  genome analysis (e.g., determining how it’s unique), it’s the resulting data sets that are accessible by doctors, specialists and others to really make sense of it all.

Presumably, Bina is referring to companies such as DNAnexus when it compares its solution to entirely cloud-based approaches. DNAnexus is another Silicon Valley startup trying to democratize genome analysis, relying on the processing power and centralized nature of the cloud to serve as a platform for analyzing and collaborating on DNA data. Another startup, St. Louis-based Appistry, has taken a somewhat different approach, building its own high-powered cloud service and developing its own algorithms specially designed for genome analysis.

In the end, it’s all about the data

Regardless of which approach a researcher takes to solving the problem of sequenced genome data (they all have unique benefits), the underlying trend driving innovation is the deluge of genome data itself. Bani Asadi said the biggest difference now compared with past efforts to analyze health data is that we have so much available. There are 30,000 fully sequenced genomes available right now, and some predict there will be 10 million in five years.

That means researchers can study DNA at a much more-granular level the previously possible, Bani Asadi said, and they can analyze findings across huge data sets to identify previously undetectable patterns. Especially with cancer, she said, each case is relatively unique, shaped by many conditions and factors. If we’re going to make significant progress on treating it, we’ll need to know exactly what’s going on in any given case and how similar cases have played out. The more data, the more accurate the diagnosis and treatment.

Feature image courtesy of Keith Edkins.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Dissecting the data: 5 issues for our digital future

It’s a noble goal, but also a necessary one. Big data does have the potential to change our lives, but to get there it’s going to take more than startups created to feed us better advertisements.

Consumer data is easy to get, and profitable

It’s not fair to call the current state of big data problematic, but it is largely focused on profit-centric technologies and techniques. That’s because as companies — especially those in the web world — realized the value they could derive from advanced data analytics, they began investing huge amounts of money in developing cutting-edge techniques for doing so. For the first time in a long time, industry is now leading the academic and scientific research communities when it comes to technological advances.

As Brenda Dietrich, IBM Fellow and vice president for business analytics for IBM Software (and former VP of IBM’s mathematical sciences division), explained to me, universities are still doing good research, but students are leaving to work at companies like Google and Facebook as soon as their graduate or Ph.D. studies are complete, often times beforehand. Research begun in universities is continued in commercial settings, generally with commercial interests guiding its direction.

And this commercial focus isn’t ideal for everyone. For example, Sultan Meghji, vice president of product strategy at Appistry, told me that many of his company’s government- and intelligence-sector customers aren’t getting what they expected out of Hadoop, and they’re looking for alternative platforms. Hadoop might well be the platform of choice for large web and commercial applications — indeed, it’s where most of those companies’ big data investments are going — but it has its limitations.

Enter federal dollars for big data

However, as John Holdren, assistant to the president and director of White House Office of Science and Technology Policy, noted during a White House press conference on Thursday afternoon, the Obama administration realized several months ago that it was seriously under-investing in big data as a strategic differentiator for the United States. He was followed by leaders from six government agencies explaining how they intend to invest their considerable resources to remedy this under-investment. That means everything from the Department of Defense, DARPA and the Department of Energy developing new techniques for storage and management, to the U.S. Geological Survey and the National Science Foundation using big data to change the way we research everything from climate science to educational techniques.

How’s it going to do all this, apart from agencies simply ramping up their own efforts? Doling out money to researchers. As Zach Lemnios, Assistant Secretary of Defense for Research & Engineering for the Department of Defense, put it, “We need your ideas.”

IBM’s Deitrich thinks increased availability of government grants can play a major role in keeping researchers in academic and scientific settings rather than bolting for big companies and big paychecks. Grants can help steer research away from targeted advertising and toward areas that will “be good … for mankind at large,” she said.

The 1,000 Genomes Project data is now freely available to researchers on Amazon's cloud.

Additionally, she said, academic researchers have been somewhat limited in what they can do because they haven’t always had easy access to meaningful data sets. With the government now pushing to open its own data sets, and as well as for collaborative research among different scientific disciplines, she thinks there’s a real opportunity for researchers to do conduct better experiments.

During the press conference, Department of Energy Office of Science Director William Brinkman expressed his agency’s need for better personnel to program its fleet of supercomputers. “Our challenge is not high-performance computing,” he said, “it’s high-performance people.” As my colleague Stacey Higginbotham has noted in the past, the ranks of Silicon Valley companies are deep with people who might be able to bring their parallel-programming prowess to supercomputing centers if the right incentives were in place.

Self-learning systems, a storage revolution and a cure for cancer?

As anyone who follows the history of technology knows, government agencies have been responsible for a large percentage of innovation over the past half century, taking credit for no less than the Internet itself. “You can track every interesting technology in the last 25 years to government spending over the past 50 years,” Appistry’s Meghji said.

Now, the government wants to turn its brainpower and money to big data. As part of its new, roughly $100-million XDATA program, DARPA Deputy Director Kaigham “Ken” Gabriel said his agency “seek[s] the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data. The DOE’s Brinkman talked about the importance of being able to store and visualize the staggering amounts of data generated daily by supercomputers, or by the second from CERN’s Large Hadron Collider.

IBM’s Dietrich also has an idea for how DARPA and the DOE might spend their big data allocations. “When one is doing certain types of analytics,” she explained, “you’re not looking at single threads of data, you tend to be pulling in multiple threads.” This makes previous storage technologies designed to make the most-accessed data the easiest to access somewhat obsolete. Instead, she said, researchers should be looking into how to store data in a manner that takes into account the other data sets typically accessed and analyzed along with any given set. “To my knowledge,” she said, “no one is looking seriously at that.”

Not surprisingly given his company’s large focus on genetic analysis, Appistry’s Meghji is particularly excited about the government promising more money and resources in that field. For one, he said, the Chinese government’s Beijing Genomics Institute probably accounts for anywhere between 25 and 50 percent of the genetics innovation right now,  and “to see the U.S. compete directly with the Chinese government is very gratifying.”

But he’s also excited about the possibility of seeing big data turned to areas in genetics other than cancer research — which is presently a very popular pastime — and generally toward advances in real-time data processing. He said the DoD and intelligence agencies are typically two to four years ahead of the rest of the world in terms of big data, and increased spending across government and science will help everyone else catch up. “It’s all about not just reacting to things you see,” he said, “but being proactive.”

Indeed, the DoD has some seriously ambitious plans in place. Assistant Secretary Lemnios explained during the press conference how previous defense research has led to technologies such as IBM’s Watson system and Apple’s Siri that are becoming part of our everyday lives. Its latest quest: utilize big data techniques to create autonomous systems that can adapt to and act on new data inputs in real time, but that know enough to know when they need to invite human input on decision-making. Scary, but cool.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011