Python is already incredibly popular among programmers of all levels in all industries — not just with web programmers (including those at Google), but also with the scientific community. Continuum’s stated mission is “developing the next generation of tools to make Python as powerful and successful for big data and business data analytics as it has been for science, engineering, and scalable computing.”

Continuum’s flagship product, Anaconda, uses Disco — a Python-based (and Nokia-developed) take on the Java-based Hadoop platform — and supports popular scientific Python libraries such as NumPy and SciPy. The company also offers a product called Wakari, which is a browser-based analytics environments that it describes as “WordPress, Github, and Youtube for science, engineering, and business data analytics.”

DARPA appears particularly interested in some of the open source efforts that Continuum is developing and sponsoring itself. These include Blaze, a technology for writing Python code that can run analytic jobs across distributed systems and different environments; Bokeh, an HTML5 data visualization library designed for big, multidimensional data; and Numba, a compiler for turning Python code into native machine code to improve computing speed.

Considering the Defense Department’s goals with the XDATA program — “seeking the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data — putting money behind a company like Continuum makes sense. Its core technologies are designed for the heavy computational needs and data-processing and visualization, but the Python foundation and commercialization efforts make them accessible by a broader array of users.

And anyone interested in how the defense and intelligence agencies are using big data technologies to further their missions really should check out our Structure: Data conference in New York next month. Not only will we be discussing the cutting edge in data analysis, but we’ll have keynote presentations by Central Intelligence Agency CTO Ira “Gus” Hunt and Samantha Ravich, co-chair of the National Commission for Review of R&D Programs for the United States Intelligence Community.

Feature image courtesy of Shutterstock user argus.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The Incredible, Growing, Commercial Hadoop Market
• The importance of putting the U and I in visualization

It’s designed with speed in mind for uses such as in-memory analytics (including on SAP’s popular HANA platform) and certain scientific workloads that require data delivery to keep up with processing speed. The faster applications can read and write data — and doing so from an in-memory cache or solid-state drives is much faster than doing so from hard drives — the sooner that processors can compute it.

And because the new instance is part of AWS’s Cluster Compute family, multiple instances are connected via a 10 GbE network for speedy server-to-server data transfer. In benchmark tests from a site called CloudHarmony in 2010, Cluster Compute instances far outperformed anything else on the market (GigaOM Pro subscription req’d) at the time. They’ve also been used to spin up clusters that can compete with traditional supercomputers in terms of sheer performance — reaching No. 102 on the lastest Top500 list with a peak speed of 354.1 teraflops.

Although, it should be noted, AWS isn’t the only game in town for users wanting this type of beefy core in order to handle their real-time data processing needs. Liquid Web’s Storm cloud service, for example, offers some high-memory, SSD-powered servers of its own at nearly $1.50 per hour less than what AWS charges (albeit with fewer cores and absent the 10 GbE backbone and list of features that comes along with the AWS platform).

Whatever the cloud, though, ever-higher-performing instances mean new classes of workloads and more business for cloud providers that offer them. Especially as big data and analytics applications pick up steam and move from batch to real-time, clouds that can handle demanding users are in a good position.

Feature image courtesy of Shutterstock user ssguy.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data third-quarter 2012
• Takeaways from the second quarter in cloud and data
• Infrastructure Q1: Cloud and big data woo enterprises

Matt Wood

He and I sat down this week at AWS’s inaugural Re: Invent conference and talked about many things, including what he’s seen in the field and where cloud-based big data efforts are headed. Here are the highlights.

The end of contstraint-based thinking

Not so long ago, computer scientists understood many of the concepts that we now call data science, but limited resources meant they were hamstrung in the types of analysis they could attempt to do. “That can be very limiting, very constraining when you’re working with data,” Wood said.

Now, however, data storage and processing resources are relatively inexpensive and abundant — so much so that they’ve actually made the concept of big data possible. Cloud computing has only made those resources cheaper and more abundant. The result, Wood said, is that people working with data are undergoing a shift from that mindset of limiting their data analysis to the resources they have available to one where they think about business needs first.

If they’re able to get past traditional notions of sampling and days-long processing times,  he added, individuals can focus their attention on what they can do because they have so many resources available. He noted how Yelp gave developers relatively free rein early on the use of Elastic MapReduce, saving them from having to formally request resources just “to see if the crazy idea [someone] had over coffee is going to play out.” Yelp was able to spot a shift in mobile traffic volume years ago and get a headstart on its mobile efforts because of that, Wood added.

Data problems aren’t just about scale

Generally speaking, Wood said, solving customers’ data problems isn’t just about figuring out how to store ever greater volumes for every cheaper prices. “You don’t have to be at a petabyte scale in order to get some insight on who’s using your social game,” he said.

In fact, access to limitless storage and processing is a solution to one problem that actually creates another. Companies want to keep all the data they generate, and that creates complexity, Wood explained. As that data piles up in various repositories — perhaps in Amazon’s S3 and DynamoDB services, as well as on some physical machines with a company’s data center — moving it from place to place in order to reuse it becomes a difficult process.

Wood said AWS built its new Data Pipeline Service in order to address this problem. Pipelines can be “arbitrarily complex,” he explained — from running a simple piece of business logic against data to running whole batches through Elastic MapReduce — but the idea is to automate the movement and processing so users don’t have to build these flows themselves and then manually run them.

The cloud isn’t just for storing tweets

People sometimes question the relevance of cloud computing for big data workloads, if only because any data generated on in-house systems has to make its way to the cloud over inherently slow connections. The bigger the dataset, the longer the upload time.

Wood said AWS is trying hard to alleviate these problems. For example, partners such as Aspera and even some open source projects enable customers to move large files at fast speeds over the internet (Wood said he’s seen consistent speeds of 700 megabits per second). This is also why AWS has eliminated data-transfer fees for inbound data, has turned on parallel uploads for large files and created its Direct Connect program with data center operators that provide dedicated connections to AWS facilities.

And if datasets are too large for all those methods, customers can just send AWS their physical disks. “We definitely receive hard drives,” Wood said.

Collaboration is the future

Once data makes its way to the cloud, it opens up entirely new methods of collaboration where researchers or even entire industries can access and work together on shared datasets too big to move around. “This sort of data space is something that’s becoming common in fields where there are very large datasets,” Wood said, citing as an example the 1000 Genomes project dataset that AWS houses.

DNAnexus’s cloud-based architecture

As we’ve covered recently, the genetics space is drooling over the promise of cloud computing. The 1000 Genomes database is only 200TB, Wood explained, but very few project leads could get the budget to store that much data and make it accessible to their peers, much less the computation power required to process it. And even in fields such as pharmaceuticals, Amazon CTO Werner Vogels told me during an earlier interview, companies are using the cloud to collaborate on certain datasets so companies don’t have to spend time and money reinventing the wheel.

No more supercomputers?

Wood seemed very impressed with the work that AWS’s high-performance computing customers have been doing on the platform — work that previously would have been done on supercomputers or other physical systems. Thanks to AWS partner Cycle Computing, he noted, the Morgridge Institute at the University of Wisconsin was able to perform 116 years worth of computing in just one week. In the past, access to that kind of power would have required waiting in line until resources opened up on a supercomputer somewhere.

The collaborative efforts Wood discussed certainly facilitate this type of extreme computation, as does AWS’s continuous efforts to beef up its instances with more and more power. Whatever users might need, from the new 250GB RAM on-demand instances to GPU-powered Cluster Compute Instances, Wood said AWS will try to provide it. Because cost sometimes matters, AWS has opened Cluster Compute Instances and Elastic MapReduce to its spot market for buying capacity on the cheap.

But whatever data-intensive workloads organizations want to run, many will always look to the cloud now. Because cloud computing and big data — Hadoop, especially — have come of age roughly in parallel with each other, Wood hypothesized, they often go hand-in-hand in people’s minds.

Feature image courtesy of Shutterstock user winui.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• It’s time for cloud security and big data to come together
• Buying into big data appliances
• 9 Companies that Pushed the Infrastructure Discussion in 2010

Autodesk has actually been moving various products and services to the cloud in some form for a few years, but the new product is different, said Grant Rochelle, the company’s senior director of manufacturing industry marketing. Most importantly, it’s cheap. Rochelle said traditional simulation software can cost between $20,000 and $100,000 per user per year, and that’s not to mention the high-performance systems necessary to run it. Many employees who need it don’t get it, and many companies are shut out from purchasing it altogether.

Simulation 360, on the other hand, costs mere thousands. For $3,200 a year, users can run a total of 120 jobs. For $7,200, they get unlimited access. For a limited time, actually, unlimited use is free. And rather than needing separate products for mechanical, fluid and thermal simulation, they’re all included in the new cloud offering.

Rochelle said the goal of Simulation 360 isn’t to move big-money customers such as large aerospace companies or defense contractors to the cloud service (“they will be the last bastion of adopting cloud for most things,” he said), but to attract entirely new users building innovative new products or doing interesting work. One customer, for example, is working to make hospital operating rooms as clean as the rooms in which microprocessors are fabricated. It’s simulating the spread of airborne germs through a hospital to prevent certain illnesses that patients contract while receiving treatment for something else.

Another customer, BioLite, has developed a product that turns the excess heat from a fire into electricity so, for example, campers or individuals in third-world countries can ensure their portable electronics are always charged. Rochelle noted that green building and heating and cooling, generally, are areas where the new cloud offering looks to open up a lot of new doors.

However, while offering this technology as a cloud service might be new, the value story is as old as cloud computing itself. Give innovators without huge IT budgets access to resources at a price point never before possible and see what happens. Maybe it’s Instagram, maybe it’s a new approach to climate modeling, maybe it’s a revolution in energy-efficient HVAC systems, but chances are we’re better off because of it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• As Devices Converge, Chip Vendors Girding For a Fight
• A near-term outlook for big data
• Flash analysis: the tech startup investment environment, Q3 2011

Aspiring heirs to the Klout throne, for example, might look to a project called STINGER currently under development at Georgia Tech University. STINGER, which stands for Spatio-Temporal Interaction Networks and Graphs Extensible Representation, is a graph-processing engine that project lead David Bader says is bigger, faster and more flexible than anything currently in use for analyzing social media connections. You provide a shared-memory computing system, and it provides an open-source tool that can help detect relationships between billions of people, places and things as those relationships change over time — even in real time.

Someone using Facebook data, for example, might write an algorithm using where people or pages would be the vertices and actions (likes, shares, wall posts, etc.) would be the graph’s edges. One relatively easy application, Bader explained, would be to analyze how activity around particular people is increasing, decreasing or changing, therefore indicating changes in their importance or the growth of new communities.

We’ll do the hard work

Writing an algorithm to perform that kind of analysis isn’t really the problem, though — it’s writing one that can scale into the billions of vertices and edges and still perform quickly enough to be useful. An algorithm that generates one false positive in a million isn’t so bad when you’re dealing with tens of thousands of items, Bader explained, but it gets to be a big problem when you’re talking about billions of items against which it’s running.

There are dozens of open source graph databases available, including popular offerings such as Neo4j and InfiniteGraph, but he said, “Our lab focuses on algorithms that run fast on massive data sets and that are more accurate than what is traditionally done in social media.”

David Bader

Bader’s team recently presented a paper detailing a social media algorithm running atop STINGER that ran 100 times faster than some previous approaches because the system stores the graph’s previous state and only performs the minimal amount of processing necessary as new edges are inserted. This is in contrast to traditional approaches that re-process the entire graph every time there’s a change.

That being said, Georgia Tech isn’t entirely alone analyzing massive amounts of social data with graph databases. Google’s Pregel had already scaled to billions of vertices and edges as of 2009, and Facebook is currently analyzing more than a billion edges using Apache Giraph (an open source, Hadoop-based Pregel implementation). But those cases — both companies are loaded with smart engineers, data scientists and powerful infrastructure — just underscore the importance of what researchers like Bader are building and releasing as open source.

Forget social media, solve real problems

But social data isn’t just useful for figuring out who’s influential on Twitter or Facebook — it also can be used to solve some real problems. Bader said he’s already used graph processing with Twitter data to determine who was leading resistance units during Egypt’s recent revolution. ”Anywhere I can look at connections between entities,” he said, “these approaches are available.”

Indeed. On Wednesday afternoon, for example, a group of researchers from the University of Alberta, University of Connecticut and University of California-Merced unveiled a new data-based method that could make it faster, easier and less expensive to root out culprits in fraud cases.

The technique uses a method called the Steiner tree to analyze the connections –social, business, familial, etc. — between the people involved in a given case of fraud. The algorithm is able to determine the shortest path between two objects, which the researchers posit is especially applicable to fraud investigations — the person with the shortest path between himself and the crime is probably the culprit (or at least a solid suspect).

The fraud researchers’ paper follows the publication in August of a method for determining the source of everything from a disease outbreak to a Twitter rumor by tracking its spread across a complex network over time. Their algorithm, the paper’s authors claim, could be particularly effective for combating cybercrime by tracking computer viruses back to their sources. The more connections (in the case of social data), or observers, a particular point has, the fewer that are needed to track down the source point.

However, all the algorithms and data frameworks in the world probably won’t make too big a difference until they’re turned into products that actually work on real-world situations. As the University of Alberta’s Ray Patterson pointed out in a press release detailing the fraudster-detection algorithm, ”It might take several years or many years before anyone picks it up. But it’s a good thing if we can point people towards what’s useful.”

Georgia Tech’s Bader said DARPA, Intel, Sandia National Laboratory and other research institutions have already used STINGER to tackle some complex data sets, and he suspects a strong commercial interest, as well. If a company is willing to take STINGER from a project into a product, it could bring the project’s scale and speed to everything from analyzing customer interactions to monitoring the changing nature of criminal networks, Bader said. Considering the desire from companies of all types to extract some meaning from social data, I have to think someone will give it a shot.

Feature image courtesy of Shutterstock user 3DProfi.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Best practices in optimizing content for social engagement
• 12 tech leaders’ resolutions for 2012

The results, detailed in a blog post on Tuesday, suggest that Google Compute Engine’s real problem right now might just be a lack of high-performance instances. Its current workhorse — an 8-core Intel Sandy Bridge instance with 30GB of memory and 22 compute units — can’t hang with the Amazon Cluster Compute Instances that Zencoder uses for its transcoding workloads. The largest of those is a 16-core dual-CPU Intel Xeon instance providing 60.5GB of memory and 88 compute units running atop a 10 Gigabit Ethernet platform.

As Zencoder ramped up the workloads, the performance differences became clear:

Compute Engine didn’t fare any better when Zencoder tested transfer speeds between the cloud storage platform and the cloud computing platform. Whereas rates between Amazon S3 and Amazon EC2 topped out at 1,458.32 Mbps, the rate between Google Cloud Storage and Google Compute Engine peaked at 202.6 Mbps. In fact, the post’s author writes, “it appears that GCS is slower than S3, and GCE transfer is slower than EC2, such that even if you’re using Google for compute, you may be better off using S3 for storage.”

While the results are interesting because they’re the first real apple-to-apples comparison I’ve seen between Compute Engine and EC2 (BuildFax cloud architect Joe Emison’s pre-release benchmarks were pulled from his Compute Engine review on InformationWeek), they need to taken as what they are. They are, as Zencoder points out, tests of a specific CPU-bound workload — the performance of which Google could improve by adding higher-powered instances — and don’t take into account the difficulties of running at massive scale — a capability Google touted when it launched Compute Engine in June.

And, the author notes, Compute Engine is generally a quality platform, “especially [with regard to] disk I/O, boot times, and consistency, which historically haven’t been EC2′s strong suit.”

This might actually be the more-important measure for most potential Compute Engine users. As GigaOM contributor James Urquhart wrote recently, “If Google can deliver a service that eliminates most of the I/O and network performance inconsistencies that AWS customers currently experience, I can guarantee you there are many major compute customers of AWS that will want to give Compute Engine a test run.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cloud
• How direct-access solutions can speed up cloud adoption
• Cloud computing 2013: how to navigate without a map

Domain expertise, statistics and HPC, unite!

The latest is Bina Technologies, which just emerged from stealth mode last week and is bringing an Apple-like business model to genomics. The company, which grew out of a research project at Stanford University, relies on its Bina Box appliance to make genome analysis faster than typically possible without the benefit of having a supercomputer and a research network on hand.

According to Bina CEO Narges Bani Asadi, who co-founded the company while completing her Ph.D. at Stanford, the appliance came about as part of a mission to solve a disconnect among the stakeholders in cancer research. Improving the analysis of cancer data required input from medical researchers, statisticians and high-performance computing experts, “but people are not speaking even the same language,” she said. While they’re all headed in the same direction, their paths rarely converge to harness peak velocity.

Bani Asadi and her team solved that problem by developing a system that merged the three areas into one. With Bina, researchers can develop analysis pipelines that are optimized at both the algorithmic and silicon levels to run optimally across a mix of CPUs, GPUs and FPGAs, all of which are present within the purpose-built box. Applications are getting what they need in order to perform their best, and Bina says results can be processed 10 to 100 times faster (hours instead of days) than running jobs on the Amazon Web Services cloud, which has proven very popular for genomics workloads thanks to its supercomputer-like performance.

That being said, a chart Bina uses to illustrate the performance difference compares the Bina Box to a single eight-core AWS instance rather than a cluster of those high-performance instances.

The Apple analogy

Bani Asadi answers the inevitable question of whether research centers will want to special appliances instead of using the cloud or generic servers by pointing to Apple. That company’s devices and computers can be a little more expensive and more difficult to tinker with than alternatives, but they’re also designed specifically with Apple’s operating system and applications in mind. It’s an analogy other companies, such as cloud computing startup Nebula, also use to justify their appliance-based businesses.

Not that Bina dismisses the cloud. If Bina’s software is the Mac OS to the Bina Box’s iMac, the Bina Cloud is the company’s iCloud. Once the box processes the raw sequencing data and compresses it into a smaller volume (up to 1,000 times smaller), the data is shipped to the Bina Cloud where it’s stored and can be easily accessed and shared. Actually, Bani Asadi said that’s where the most-innovative research likely will take place. While Bina’s appliance handles the necessary first steps of  genome analysis (e.g., determining how it’s unique), it’s the resulting data sets that are accessible by doctors, specialists and others to really make sense of it all.

Presumably, Bina is referring to companies such as DNAnexus when it compares its solution to entirely cloud-based approaches. DNAnexus is another Silicon Valley startup trying to democratize genome analysis, relying on the processing power and centralized nature of the cloud to serve as a platform for analyzing and collaborating on DNA data. Another startup, St. Louis-based Appistry, has taken a somewhat different approach, building its own high-powered cloud service and developing its own algorithms specially designed for genome analysis.

In the end, it’s all about the data

Regardless of which approach a researcher takes to solving the problem of sequenced genome data (they all have unique benefits), the underlying trend driving innovation is the deluge of genome data itself. Bani Asadi said the biggest difference now compared with past efforts to analyze health data is that we have so much available. There are 30,000 fully sequenced genomes available right now, and some predict there will be 10 million in five years.

That means researchers can study DNA at a much more-granular level the previously possible, Bani Asadi said, and they can analyze findings across huge data sets to identify previously undetectable patterns. Especially with cancer, she said, each case is relatively unique, shaped by many conditions and factors. If we’re going to make significant progress on treating it, we’ll need to know exactly what’s going on in any given case and how similar cases have played out. The more data, the more accurate the diagnosis and treatment.

Feature image courtesy of Keith Edkins.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Dissecting the data: 5 issues for our digital future

For the San Francisco-based Metamarkets (see disclosure), the Series B round brings its total to $23.5 million, and represents some serious confidence in the company’s cloud-based analytics platform. Investors should be confident: the company has a handful of customers, but they’re rather large, and it has already spurned acquisition offers from some household names in the IT world, including Twitter.

ParAccel is in another league altogether, having now raised nearly $100 million over the past several years, and having just completed a first quarter that saw 500 percent year-over-year revenue growth. It’s one of the best-known independent analytic database providers around after Greenplum, Netezza and Vertica all got snatched up by large vendors in the past few years. Last summer, Amazon came on board as a strategic investor, a move rife with possibility considering persistent rumors of an Amazon Web Services analytic offering.

However, big data isn’t constrained to companies building the technology. Often times, as with DataPop, it’s the consumers of big data technologies that are the most interesting. At just under $9 million, it’s hardly the highest-backed user of big data, even in the marketing space, but the story is the same across the board. In theory, big data means more-accurate and dynamic ad targeting, which means more money for everyone.

Terascala plays in an entirely different field — the high-performance computing field — where the speed of big data systems becomes critical. Terascala and companies of its ilk don’t analyzing anything, rather they feed enormous scientific (and other) data sets to high-performance processors without creating a bottleneck. That means their research, government, media and financial services users can do existing analyses much faster, and can even do entirely new types of analysis that used to be slowed by a lack of performance and a lack of analytic tools (e.g., Hadoop).

When buzzwords reach a certain level of ubiquity, they start to mean both everything and nothing, which is arguably the case for big data right now. But I don’t see it as too big a problem. I see it as a new understanding of the power of data, which — harnessed correctly — is immense. For investors, it’s not a question of whether to put money behind big data, but of figuring out which of the dozens pitching themselves as big data companies are actually doing it right.

Image courtesy of Flickr user kaibara87.

Disclosure: Metamarkets is a portfolio company of True Ventures, which is also an investor in GigaOM. Om Malik is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Will the Real-Time Web Bring High Performance to a System Near You?
• How data warehousing is now a cost-effective solution for businesses

The problem for computational chemists and biologists is there’s a trade-off between accuracy and speed. This  ”Naga” compute environment built by Cycle atop the Amazon cloud eased that tradeoff, said Ramy Farid, president of New York-based Schrödinger, which specializes in computational drug design.

“We’ve got these really accurate methods but they would take months on a normal cluster. The problem is we want to do the best possible science fast,” Farid said in an interview this week.

That’s where Cycle Computing comes in. The company has made its name building high-performance computing atop AWS infrastructure and has previously deployed 10,000 and 30,000 core clusters on the cloud. For Schrödinger, it upped the ante to 50,000 cores. The alternative in this case would be for Schrödinger to build its own 50,000-core cluster or log time on a supercomputer, said Cycle Computing CEO Jason Stowe.

HPC for rent

“Practically speaking, the latter is impractical for a for-profit company and is generally restrictive. If you’re an academic wanting time on the San Diego Super Computer, for example, you’ll have months of wait time to get approved and even then you get a limited-time window — so if something with your software is not working at that time, you’re out of luck.” And, big supercomputers don’t tend to run the kinds of software these companies want to run. “The beauty of the cloud is it runs your flavor of Linux and other software,” Farid said.

On the other hand, building a 50,000 core cluster could easily cost $20 million to $30 million, he said. The Schrödinger project, by contrast, cost about $4,850 per hour to run.

For this trial, the cluster had access to all regions of AWS and used all of them in some capacity. The application used the various EC2 APIs to provision the resources.

“All the compound data for analysis was uploaded into S3 [Amazon's Simple Storage System]. The cluster was provisioned along side it and grabbed data from S3 to run the calculations and then pushed it back into S3,” said Crowe, who will be talking about the implementation Thursday at an Amazon event in New York. The application also took advantage of some Amazon IP address and DNS capabilities.

Finding a needle in a very big haystack

Farid could not talk much about the specific research purpose other than to say the goal was to find compounds that could be developed into drugs that fight a type of cancer.  The use of the huge cluster enabled Schrödinger to use the more accurate version of its Glide software — in the past it would have had to use the less accurate screen and perhaps miss some compounds that could be extremely useful.

“The problem they’re solving is amazing. It’s like the target is a lock and the confirmation is a key — what Ramy’s software does is let you simulate 21 million keys potentially matching that lock. There are typically a number of false negatives and the reason is you can’t sample all the orientations [of key to lock] properly so it will not show a match that could actually be a match. You could miss amazing drugs that could have an impact,” Crowe said.

The use of this huge cluster enabled Schrödinger to run its much more accurate, but much more compute-intensive version of its screening software and find compound candidates the other software may have missed.

The result of the three-hour run? “We identified  a number of compounds that we will purchase and test,” Farid said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 2012: The Hadoop infrastructure market booms
• A near-term outlook for big data
• NewNet Q2: Google closes the quarter with a bang

It’s a noble goal, but also a necessary one. Big data does have the potential to change our lives, but to get there it’s going to take more than startups created to feed us better advertisements.

Consumer data is easy to get, and profitable

It’s not fair to call the current state of big data problematic, but it is largely focused on profit-centric technologies and techniques. That’s because as companies — especially those in the web world — realized the value they could derive from advanced data analytics, they began investing huge amounts of money in developing cutting-edge techniques for doing so. For the first time in a long time, industry is now leading the academic and scientific research communities when it comes to technological advances.

As Brenda Dietrich, IBM Fellow and vice president for business analytics for IBM Software (and former VP of IBM’s mathematical sciences division), explained to me, universities are still doing good research, but students are leaving to work at companies like Google and Facebook as soon as their graduate or Ph.D. studies are complete, often times beforehand. Research begun in universities is continued in commercial settings, generally with commercial interests guiding its direction.

And this commercial focus isn’t ideal for everyone. For example, Sultan Meghji, vice president of product strategy at Appistry, told me that many of his company’s government- and intelligence-sector customers aren’t getting what they expected out of Hadoop, and they’re looking for alternative platforms. Hadoop might well be the platform of choice for large web and commercial applications — indeed, it’s where most of those companies’ big data investments are going — but it has its limitations.

Enter federal dollars for big data

However, as John Holdren, assistant to the president and director of White House Office of Science and Technology Policy, noted during a White House press conference on Thursday afternoon, the Obama administration realized several months ago that it was seriously under-investing in big data as a strategic differentiator for the United States. He was followed by leaders from six government agencies explaining how they intend to invest their considerable resources to remedy this under-investment. That means everything from the Department of Defense, DARPA and the Department of Energy developing new techniques for storage and management, to the U.S. Geological Survey and the National Science Foundation using big data to change the way we research everything from climate science to educational techniques.

How’s it going to do all this, apart from agencies simply ramping up their own efforts? Doling out money to researchers. As Zach Lemnios, Assistant Secretary of Defense for Research & Engineering for the Department of Defense, put it, “We need your ideas.”

IBM’s Deitrich thinks increased availability of government grants can play a major role in keeping researchers in academic and scientific settings rather than bolting for big companies and big paychecks. Grants can help steer research away from targeted advertising and toward areas that will “be good … for mankind at large,” she said.

The 1,000 Genomes Project data is now freely available to researchers on Amazon's cloud.

Additionally, she said, academic researchers have been somewhat limited in what they can do because they haven’t always had easy access to meaningful data sets. With the government now pushing to open its own data sets, and as well as for collaborative research among different scientific disciplines, she thinks there’s a real opportunity for researchers to do conduct better experiments.

During the press conference, Department of Energy Office of Science Director William Brinkman expressed his agency’s need for better personnel to program its fleet of supercomputers. “Our challenge is not high-performance computing,” he said, “it’s high-performance people.” As my colleague Stacey Higginbotham has noted in the past, the ranks of Silicon Valley companies are deep with people who might be able to bring their parallel-programming prowess to supercomputing centers if the right incentives were in place.

Self-learning systems, a storage revolution and a cure for cancer?

As anyone who follows the history of technology knows, government agencies have been responsible for a large percentage of innovation over the past half century, taking credit for no less than the Internet itself. “You can track every interesting technology in the last 25 years to government spending over the past 50 years,” Appistry’s Meghji said.

Now, the government wants to turn its brainpower and money to big data. As part of its new, roughly $100-million XDATA program, DARPA Deputy Director Kaigham “Ken” Gabriel said his agency “seek[s] the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data. The DOE’s Brinkman talked about the importance of being able to store and visualize the staggering amounts of data generated daily by supercomputers, or by the second from CERN’s Large Hadron Collider.

IBM’s Dietrich also has an idea for how DARPA and the DOE might spend their big data allocations. “When one is doing certain types of analytics,” she explained, “you’re not looking at single threads of data, you tend to be pulling in multiple threads.” This makes previous storage technologies designed to make the most-accessed data the easiest to access somewhat obsolete. Instead, she said, researchers should be looking into how to store data in a manner that takes into account the other data sets typically accessed and analyzed along with any given set. “To my knowledge,” she said, “no one is looking seriously at that.”

Not surprisingly given his company’s large focus on genetic analysis, Appistry’s Meghji is particularly excited about the government promising more money and resources in that field. For one, he said, the Chinese government’s Beijing Genomics Institute probably accounts for anywhere between 25 and 50 percent of the genetics innovation right now,  and “to see the U.S. compete directly with the Chinese government is very gratifying.”

But he’s also excited about the possibility of seeing big data turned to areas in genetics other than cancer research — which is presently a very popular pastime — and generally toward advances in real-time data processing. He said the DoD and intelligence agencies are typically two to four years ahead of the rest of the world in terms of big data, and increased spending across government and science will help everyone else catch up. “It’s all about not just reacting to things you see,” he said, “but being proactive.”

Indeed, the DoD has some seriously ambitious plans in place. Assistant Secretary Lemnios explained during the press conference how previous defense research has led to technologies such as IBM’s Watson system and Apple’s Siri that are becoming part of our everyday lives. Its latest quest: utilize big data techniques to create autonomous systems that can adapt to and act on new data inputs in real time, but that know enough to know when they need to invite human input on decision-making. Scary, but cool.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011