At its core, Ayasdi’s product, a cloud-based service called the Insight Discovery Platform, is a mix of distributed computing, machine learning and user experience technologies. It processes data, discovers the correlations between data points, and then displays the results in a stunning visualization that’s essentially a map of the dataset and the connections between every point within it. In fact, Ayasdi is based on research into the field of topological data analysis, which Co-founder and President Gunnar Carlsson describes a quest to present data as intuitively as possible based solely on the similarity of (or distance between, in a topological sense) the data points.

It might sound complicated, but anyone familiar with the idea of social networks can probably understand how Ayasdi’s software works. Our social graphs are maps of relationships between ourselves and all of our connections, in which clusters of names (called “nodes” in computer science parlance) and connections (called “edges” in that same parlance) illustrate who’s connected to whom. Assuming someone is connected with people from different parts of his or her life, the graph will likely bear this out visually via, for example, distinct clusters of professional, personal and academic connections.

My LinkedIn social graph (via LinkedIn Labs). Generally speaking, blue is cloud computing, green is big data, dark orange is co-workers/other journalists, light orange is law school, and purple is a former employer.

Of course, the algorithms that decipher these connections don’t know how everyone is connected, just that they are. When we see our social graphs, it’s up to us to figure out why things shape up the way they do and whether that’s significant.

In essence, Ayasdi does builds and visualizes social graphs for any type of data, although the datasets it works with are much more complicated than social data. In the demonstration I saw, for example, the software analyzed medical data from 272 cancer patients covering 25,000 different genetic markers. In a few seconds, the software’s hundreds of machine-learning algorithms analyzed all those variables and created a map showing its own distinct set of clusters.

When there are thousands of variables and many, many more possible combinations in play, Co-founder and CEO Gurjeet Singh explained, “There are so many questions to ask that you don’t have the time to ask them all.” He added, ”It doesn’t even make sense to think about where to start your analysis.”

Ayasdi’s maps look like this (although no two will ever look the same). If a researcher were looking at the map, he or she would see where it might be best to start looking for answers:

In the case of cancer data, that red cluster might represent a group of cancer survivors. As researchers dig into this area further, they might find, for example, that none of these survivors underwent chemotherapy and all share a rare genetic that might make them particularly well suited to fight off cancer cells. And while Ayasdi is working primarily with medical and pharmaceutical researchers — and claims users have already uncovered new insights even from old datasets — the company is also working with organizations involved in detecting financial fraud and terrorist activity, and those trying to discover new oil fields.

In fact, readers of Wired might have gotten a sneak peek at Ayasdi last year in reporter Jeff Beckham’s post about a project by then-Ayasdi-intern Muthu Alagappan. At the MIT Sloan Sports Analytics Conference, he demonstrated how the software was able to detect statistical patterns among professional basketball players. The resulting map showed how professional basketball players’ skills actually break down into 13 rather-defined classifications — which Alagappan then labeled — beyond the five positions by which players are usually classified.

As I explained in November while covering startup company BeyondCore — which uses different technology to do achieve a similar end — the beauty of  what Ayasdi does is that users can see the correlations without having to think up and ask the questions that would expose those connections. That, of course, has been the modus operandi for business-intelligence applications and other analytics software since the beginning of time: Users write queries or otherwise ask questions of datasets, and the the software only deals with the variables it has been asked to analyze.

If a relationship is too hidden for the naked eye to see or too complex for a researcher or analyst to really consider, the software likely won’t be asked to expose it. Software like Ayasdi and BeyondCore (the CEOs of both companies will be presenting at Structure: Data in March, by the way) aims to eliminate much of that guesswork, and focus their subject-matter expertise on actually figuring out why the data points are connected as they are.

“We don’t necessarily need to treat computers like dumb question-answering machines,” Singh said. “We can actually make them do a lot more work.”

And as cutting-edge as this might sound, the techies out there should be interested to know that Ayasdi and its underlying technology are hardly new. The company has been in stealth mode since 2008, and is the result of 12 years of research among DARPA, Stanford University (where Carlsson moonlights as a math professor) and the National Science Foundation. Still, for anyone who spends their days writing SQL queries, writing segmentation algorithms or manipulating BI applications, the idea of having software mitigate much of that discovery work might seem futuristic.

It might even seem too good to be true. When I asked Carlsson how he addresses this mindset among skeptics, he suggested they at least give the software a try before dismissing it. As a comparison, he pointed to how people are able to spot a basketball in a garage by seeing everything in one shot without having to send a beam in every direction. “In a sense,” he said, “you could view the human visual system as too good to be true.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• Dissecting the data: 5 issues for our digital future
• How direct-access solutions can speed up cloud adoption

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Who and what to watch in the new era of the living room
• CES 2012: a recap and analysis

It’s a noble goal, but also a necessary one. Big data does have the potential to change our lives, but to get there it’s going to take more than startups created to feed us better advertisements.

Consumer data is easy to get, and profitable

It’s not fair to call the current state of big data problematic, but it is largely focused on profit-centric technologies and techniques. That’s because as companies — especially those in the web world — realized the value they could derive from advanced data analytics, they began investing huge amounts of money in developing cutting-edge techniques for doing so. For the first time in a long time, industry is now leading the academic and scientific research communities when it comes to technological advances.

As Brenda Dietrich, IBM Fellow and vice president for business analytics for IBM Software (and former VP of IBM’s mathematical sciences division), explained to me, universities are still doing good research, but students are leaving to work at companies like Google and Facebook as soon as their graduate or Ph.D. studies are complete, often times beforehand. Research begun in universities is continued in commercial settings, generally with commercial interests guiding its direction.

And this commercial focus isn’t ideal for everyone. For example, Sultan Meghji, vice president of product strategy at Appistry, told me that many of his company’s government- and intelligence-sector customers aren’t getting what they expected out of Hadoop, and they’re looking for alternative platforms. Hadoop might well be the platform of choice for large web and commercial applications — indeed, it’s where most of those companies’ big data investments are going — but it has its limitations.

Enter federal dollars for big data

However, as John Holdren, assistant to the president and director of White House Office of Science and Technology Policy, noted during a White House press conference on Thursday afternoon, the Obama administration realized several months ago that it was seriously under-investing in big data as a strategic differentiator for the United States. He was followed by leaders from six government agencies explaining how they intend to invest their considerable resources to remedy this under-investment. That means everything from the Department of Defense, DARPA and the Department of Energy developing new techniques for storage and management, to the U.S. Geological Survey and the National Science Foundation using big data to change the way we research everything from climate science to educational techniques.

How’s it going to do all this, apart from agencies simply ramping up their own efforts? Doling out money to researchers. As Zach Lemnios, Assistant Secretary of Defense for Research & Engineering for the Department of Defense, put it, “We need your ideas.”

IBM’s Deitrich thinks increased availability of government grants can play a major role in keeping researchers in academic and scientific settings rather than bolting for big companies and big paychecks. Grants can help steer research away from targeted advertising and toward areas that will “be good … for mankind at large,” she said.

The 1,000 Genomes Project data is now freely available to researchers on Amazon's cloud.

Additionally, she said, academic researchers have been somewhat limited in what they can do because they haven’t always had easy access to meaningful data sets. With the government now pushing to open its own data sets, and as well as for collaborative research among different scientific disciplines, she thinks there’s a real opportunity for researchers to do conduct better experiments.

During the press conference, Department of Energy Office of Science Director William Brinkman expressed his agency’s need for better personnel to program its fleet of supercomputers. “Our challenge is not high-performance computing,” he said, “it’s high-performance people.” As my colleague Stacey Higginbotham has noted in the past, the ranks of Silicon Valley companies are deep with people who might be able to bring their parallel-programming prowess to supercomputing centers if the right incentives were in place.

Self-learning systems, a storage revolution and a cure for cancer?

As anyone who follows the history of technology knows, government agencies have been responsible for a large percentage of innovation over the past half century, taking credit for no less than the Internet itself. “You can track every interesting technology in the last 25 years to government spending over the past 50 years,” Appistry’s Meghji said.

Now, the government wants to turn its brainpower and money to big data. As part of its new, roughly $100-million XDATA program, DARPA Deputy Director Kaigham “Ken” Gabriel said his agency “seek[s] the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data. The DOE’s Brinkman talked about the importance of being able to store and visualize the staggering amounts of data generated daily by supercomputers, or by the second from CERN’s Large Hadron Collider.

IBM’s Dietrich also has an idea for how DARPA and the DOE might spend their big data allocations. “When one is doing certain types of analytics,” she explained, “you’re not looking at single threads of data, you tend to be pulling in multiple threads.” This makes previous storage technologies designed to make the most-accessed data the easiest to access somewhat obsolete. Instead, she said, researchers should be looking into how to store data in a manner that takes into account the other data sets typically accessed and analyzed along with any given set. “To my knowledge,” she said, “no one is looking seriously at that.”

Not surprisingly given his company’s large focus on genetic analysis, Appistry’s Meghji is particularly excited about the government promising more money and resources in that field. For one, he said, the Chinese government’s Beijing Genomics Institute probably accounts for anywhere between 25 and 50 percent of the genetics innovation right now,  and “to see the U.S. compete directly with the Chinese government is very gratifying.”

But he’s also excited about the possibility of seeing big data turned to areas in genetics other than cancer research — which is presently a very popular pastime — and generally toward advances in real-time data processing. He said the DoD and intelligence agencies are typically two to four years ahead of the rest of the world in terms of big data, and increased spending across government and science will help everyone else catch up. “It’s all about not just reacting to things you see,” he said, “but being proactive.”

Indeed, the DoD has some seriously ambitious plans in place. Assistant Secretary Lemnios explained during the press conference how previous defense research has led to technologies such as IBM’s Watson system and Apple’s Siri that are becoming part of our everyday lives. Its latest quest: utilize big data techniques to create autonomous systems that can adapt to and act on new data inputs in real time, but that know enough to know when they need to invite human input on decision-making. Scary, but cool.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011

That’s the premise behind the Autonomous Space Capture Challenge sponsored by DARPA and NASA and run by the MIT Space Systems Laboratory, TopCoder, and Aurora Flight Sciences. The programmable tournament will invite coders to write a computer program that enables a satellite to dock with a space object that is tumbling through space. The competition builds off the work of DARPA’s Phoenix Program, which seeks to send up nano-satellites that can attach to non-functioning satellites and re-harvest them as part of a new space system.

Programmers will need to figure the best way for the nano-satellites to hook up with the tumbling failed satellites, which are in geosynchronous orbit about 22,000 miles above the earth. The best submissions will get tested using real satellites in zero gravity aboard the International Space Station’s SPHERES satellite research platform.

Jacob Katz, a doctoral student at MIT who works on the SPHERES satellite laboratory, said they’ve done some limited testing of docking of satellites on the ISS under controlled environments. This will be a full test of the range of maneuvers available to a nano-satellite.

“In this challenge, you have no advance knowledge of how it will be rotating. We’re pushing the limits of what we can do with SPHERES and we hope to break new ground with this challenge,” Katz said.

The competition officially begins on March 28 and will include four week-long rounds of submissions. The top five submissions each week will be published so competitors can build off the work for the next week. The winning submissions will be tested on the space station in late May.

There is no big prize money payout for the winning teams. Instead, the top teams will get up to $1,000 to travel to MIT to see their algorithms in action. This is more about the challenge of solving a tough problem and helping NASA and DARPA get better results than if they tried to do the work all in-house.

Ira Heffnan, general counsel for TopCoder, said it’s unclear how many teams will enter the competition. But he said TopCoder, which has a community of about 400,000 members, often runs marathon competitions that attract hundreds, sometimes thousands of submissions.

The competition highlights the work of companies like TopCoder and Kaggle, which have taken the idea of crowdsourced competitions and hackathons and turned them into big businesses. TopCoder was founded in 2001 and used competitions as part of its consulting business. It later evolved into a full competition platform and now does 40 to 100 competitions a week for enterprise clients, with a payout of $130,000 on average weekly. TopCoder previously managed MIT’s Zero Robotics technical challenges, which involved high school and middle school students creating programs for the SPHERES satellites.

It’s a pretty cool chance to test your skills and see if a program can work in space. I don’t have the chops to enter but if I did, I’d love an opportunity to take on the challenge of space, the final frontier.

To apply, visit www.zerorobotics.org and submit an application form for the competition.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Examining the rise of crowd labor platforms in 2012
• Analyzing the wearable computing market
• Defining work in the digital age: an analysis by GigaOM Pro

Tribogenics, which spun out of DARPA-backed project at UCLA, announced today it has raised $2.5 million from Flywheel Ventures and other angels to build its portable X-ray machines, which should be available in the next year or two. The company said it’s hoping to disrupt the existing $12 billion X-ray machine industry, improving the way other existing industries such as mining, medical devices and security screening lean on the technology and enabling much broader use of X-rays by consumers.

The technology is built off a discovery made by UCLA researchers Carlos Camara, Juan Escobar, Jonathan Hird, and Seth Putterman who found they could create X-Rays bright enough to produce images from peeling adhesive tape. They were able to recreate this tribolelectrification by using an actuator that brings an epoxy surface in and out of contact with a silicone membrane. This ionizes the air and when captured in a vacuum, can create X-ray radiation. This approaches eliminates the need for high voltage, which has previously limited how portable and small other X-ray machines can get.

What Tribogenics has done is turn this whole process into a handheld product called Pocket XRF about the size of thick iPhone. It doesn’t create an image like medical X-ray machines. Instead, it’s designed to send a burst of X-rays into an object and stir up the atoms inside. Then it reads the various fingerprints of the materials inside and presents the results on a graph. That means a jeweler can tell what metals went into a ring or a safety inspector can see the lead content in a product. Miners can see if there are precious metals in a sample. And security screens can inspect objects quickly.

Dale Fox, a mobile pioneer and serial entrepreneur, helped co-found Tribogenics and now serves as its CEO. He said Pocket XRF will make its biggest impact initially in precious metals and hazardous materials, making the discovery and detection process easier and cheaper. He said the products will sell for less than $10,000 a piece initially before dropping down to a much more consumer-friendly price. Ultimately, he said Pocket XRF has the potential to become a consumer tool that brings some of the forward looking features of Star Trek’s fictional Tricorder to the masses:

“It’s like when we made the move from vacuum to transistors in electronics; when you made that jump, the power requirements went down, the devices got smaller, lower cost and more rugged. It opened up an explosion in electronics.”

I’m still getting my head around the technology and am trying to imagine how many uses we can find for it on a daily basis. But it’s a reminder of how technology is moving forward and how many sensors we can pack into increasingly smaller and smaller devices. In the coming years, we are going to be surrounded by an array of lightweight sensors that can do an astonishing amount of tasks, a lot of it mundane but some things very powerful as well.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Analyzing the wearable computing market
• 12 tech leaders’ resolutions for 2012
• GigaOM Research highs and lows from CES 2013

“I try to have a first  [investment] round that has a 50/50 mortality rate,” said Joy at MIT’s Emerging Technologies Conference on Wednesday. So essentially the early stage startups he likes to back have a 50-percent chance of not working. “Now a 90-percent mortality rate is not a good use of my time. That’s something for the government to back,” added Joy.

Joy has 10 to 12 startups he works with at anyone time, including companies like Siluria, which converts natural gas into ethylene (a key ingredient in plastics) and Renmatix, which makes sugar from biomass that can be turned into biofuel and biochemicals. Joy joined Kleiner six years ago after starting Sun Microsystems in the early ’80s.

The switch from IT to greentech has been “a learning experience,” and Joy says he’s learned enough about physics and chemistry “to not be too dangerous to our [firm's] financial future.” Joy explained: “I felt like I’d done enough with computer industry. So I was happy to be allowed to pioneer for us in this area.”

Another key way to work with failure, is to fail cheaply. Joy says Kleiner will spend a couple million dollars backing an early stage risky company to see if they can help de-risk the investment and add value. But if the company ends up taking more like $20 million to fail, then that’s not good economics for them. Entrepreneur Bill Gross has explained the same thing before.

To find his investments, Joy said the Kleiner team thought about grand challenges that it wanted to solve, like desalinating water for 10 cents per cubic meter — today desalinating water costs closer to $1 per cubic liter, said Joy. Joy would then think about the best way to get to that goal (in the desalination problem, he points to using thermal energy instead of electricity) and then would start looking for scientists that are working on those innovations. Another way is to use the new tools that have been developed in the last decade — carbon nano tubes, robotics, ionic liquids — and apply those to older problems.

When looking at competing technologies, Joy says he uses the “secretary algorithm.” When you hire a new secretary, you can’t just pick the first one you interview because she seems pretty good — you have to get a sense of the quality of the population and then pick the one that stands out, said Joy.

At the end of the day, though, one of the problems with tackling high-risk problems and backing high-risk companies, is that some of these problems and innovations just aren’t scientifically possible. So it’s one of Joy’s jobs to determine at an early point if the company can’t be taken any farther with Kleiner’s funds. That’s the fail early part.

Image courtesy of hans.gerwitz.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Green IT’s Q4 Winners: Wind Power, Solar Power, Smart Energy
• Analyzing the wearable computing market
• Ups and downs for cleantech in Q1

One of the novelties of mc10′s technology, is that it has so many applications. Medical devices (flexible sensors and surgery tools), clothing with embedded electronics for soldiers or fashion, automotive lighting, or energy and solar technology, like flexible solar panels and movement-powered materials. Yep, picture joggers of the future charging their iPods via their hoodies, and mc1o has an R&D partnership with Reebok for athletic wear.

I first heard about mc10 when the startup won backing, along with researchers from University of Illinois, from the Department of Energy’s Advanced Research Projects Agency (ARPA-E), which is a DOE program modeled after the Department of Defense’s DARPA program. The idea of ARPA-E is to award small grants to early stage, high-risk projects. mc10 and its research group received $1.71 million to make flexible nano-structured electronics that can convert waste heat into electricity.

At the time of its ARPA-E grant, mc10 said the project involved technical risks, making it hard for them to raises VC funds at that stage. Looks like either those risks were abated, or Braemer and mc10′s other investors, which include North Bridge Venture Partners, Osage University Partners, and Terawatt Ventures, are now willing to swallow the risks. The market size is large enough — scaling up and getting costs low will be the hurdle.

As devices get smaller, and sensors get embedded on everything from pill cap bottles to industrial machinery, new form factors for electronics will be needed. Batteries are going through the same stage of innovation, and ultra-thin and uniquely shaped batteries are being developed.

Image courtesy of pt.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The opportunities for the Internet and clean power
• Cleantech and investment in 2013
• Analyzing the wearable computing market

The potential efficiency might not sound breathtaking considering that the National Renewable Energy Laboratory and the Defense Advanced Research Projects Agency both announced last year that they had produced cells that achieved more than 40 percent efficiency in the lab. But lab efficiencies and production efficiencies are not the same thing, and the highest efficiencies for mass-produced solar cells hover around 22 percent. SunPower Corp., which introduced 22-percent efficiency monocrystalline cells in 2007, last year announced it had produced a prototype with 23.4 percent efficiency, which it expects to launch commercially next year.

RoseStreet thinks its new hybrid cell – which “marries low-cost nitride thin film with the massive infrastructure of silicon solar cells,” as CEO Bob Forcier said in the press release – could be cheaper than crystalline silicon cells. The nitride film, the same material used in solid-state lighting and blue lasers, enables the cells to make use of more of the light spectrum, Forcier told us.

The company plans to use a fabless model, manufacturing the cells through a partner, and also plans to license some of its technology, Forcier added. RoseStreet hopes to reach costs of less than $1.50 per watt, or 10 cents per kilowatt-hour, by 2014. The company plans to target applications with space constraints, such as industrial rooftops and mobile devices.

RoseStreet will certainly have competition. Other technologies are also racing toward higher efficiencies, with six companies – Suntech Power, Sunovia Energy Technologies, IMEC, Q-Cells, Fraunhofer Institute for Solar Energy Systems and Oerlikon – last week announcing world-record-breaking efficiencies for different types of crystalline silicon, thin film and multijunction solar cells.

And RoseStreet will have its work cut out. Many companies developing new technologies end up running into complications that result in delays. So far, at least, Forcier said he doesn’t foresee any problems with getting or processing the nitride material that it plans to use. “Nitrides are pretty pervasive in other parts of the world . . . so we see it as a scaleable technology on a worldwide basis,” he said. “We have no concerns about sourcing.”

In any case, it’s a good sign that the company expects to reach production next year. If RoseStreet really can reach such high efficiencies at a lower cost than conventional crystalline silicon, and can manufacture the cells in mass quantities, it should have no trouble finding a market.

The company has been developing higher-efficiency solar technology since at least 2004, when it licensed multiband technology from the Lawrence Berkeley National Laboratory. The company has formed partnerships with Los Alamos National Laboratory and Sumitomo Chemical Co., which also invested in RoseStreet.

The company already operates a 500-kW pilot line in Phoenix and last year launched a solar-cell development center in the same city. The company plans to seek more funding – technically its Series A round – in the first quarter of 2010, Forcier said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cleantech and investment in 2013
• The race for cost-effective and efficient solar power
• After Solyndra: analyzing the solar industry

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cleantech and investment in 2013
• Analyzing the wearable computing market
• The race for cost-effective and efficient solar power