The Defense Advanced Research Projects Agency has invested $3 million in Continuum Analytics, an Austin, Texas-based company that’s commercializing some popular methods for doing big data using the Python programming language. The investment comes out of DARPA’s XDATA fund, a $100 million program announced in 2012 that aims to develop techniques and software for analyzing large volumes of semi-structured and unstructured data.
Python is already incredibly popular among programmers of all levels in all industries — not just with web programmers (including those at Google), but also with the scientific community. Continuum’s stated mission is “developing the next generation of tools to make Python as powerful and successful for big data and business data analytics as it has been for science, engineering, and scalable computing.”
Continuum’s flagship product, Anaconda, uses Disco — a Python-based (and Nokia-developed) take on the Java-based Hadoop platform — and supports popular scientific Python libraries such as NumPy and SciPy. The company also offers a product called Wakari, which is a browser-based analytics environments that it describes as “WordPress, Github, and Youtube for science, engineering, and business data analytics.”
DARPA appears particularly interested in some of the open source efforts that Continuum is developing and sponsoring itself. These include Blaze, a technology for writing Python code that can run analytic jobs across distributed systems and different environments; Bokeh, an HTML5 data visualization library designed for big, multidimensional data; and Numba, a compiler for turning Python code into native machine code to improve computing speed.
Considering the Defense Department’s goals with the XDATA program — “seeking the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data — putting money behind a company like Continuum makes sense. Its core technologies are designed for the heavy computational needs and data-processing and visualization, but the Python foundation and commercialization efforts make them accessible by a broader array of users.
And anyone interested in how the defense and intelligence agencies are using big data technologies to further their missions really should check out our Structure: Data conference in New York next month. Not only will we be discussing the cutting edge in data analysis, but we’ll have keynote presentations by Central Intelligence Agency CTO Ira “Gus” Hunt and Samantha Ravich, co-chair of the National Commission for Review of R&D Programs for the United States Intelligence Community.
Feature image courtesy of Shutterstock user argus.