Matt Wood

He and I sat down this week at AWS’s inaugural Re: Invent conference and talked about many things, including what he’s seen in the field and where cloud-based big data efforts are headed. Here are the highlights.

The end of contstraint-based thinking

Not so long ago, computer scientists understood many of the concepts that we now call data science, but limited resources meant they were hamstrung in the types of analysis they could attempt to do. “That can be very limiting, very constraining when you’re working with data,” Wood said.

Now, however, data storage and processing resources are relatively inexpensive and abundant — so much so that they’ve actually made the concept of big data possible. Cloud computing has only made those resources cheaper and more abundant. The result, Wood said, is that people working with data are undergoing a shift from that mindset of limiting their data analysis to the resources they have available to one where they think about business needs first.

If they’re able to get past traditional notions of sampling and days-long processing times,  he added, individuals can focus their attention on what they can do because they have so many resources available. He noted how Yelp gave developers relatively free rein early on the use of Elastic MapReduce, saving them from having to formally request resources just “to see if the crazy idea [someone] had over coffee is going to play out.” Yelp was able to spot a shift in mobile traffic volume years ago and get a headstart on its mobile efforts because of that, Wood added.

Data problems aren’t just about scale

Generally speaking, Wood said, solving customers’ data problems isn’t just about figuring out how to store ever greater volumes for every cheaper prices. “You don’t have to be at a petabyte scale in order to get some insight on who’s using your social game,” he said.

In fact, access to limitless storage and processing is a solution to one problem that actually creates another. Companies want to keep all the data they generate, and that creates complexity, Wood explained. As that data piles up in various repositories — perhaps in Amazon’s S3 and DynamoDB services, as well as on some physical machines with a company’s data center — moving it from place to place in order to reuse it becomes a difficult process.

Wood said AWS built its new Data Pipeline Service in order to address this problem. Pipelines can be “arbitrarily complex,” he explained — from running a simple piece of business logic against data to running whole batches through Elastic MapReduce — but the idea is to automate the movement and processing so users don’t have to build these flows themselves and then manually run them.

The cloud isn’t just for storing tweets

People sometimes question the relevance of cloud computing for big data workloads, if only because any data generated on in-house systems has to make its way to the cloud over inherently slow connections. The bigger the dataset, the longer the upload time.

Wood said AWS is trying hard to alleviate these problems. For example, partners such as Aspera and even some open source projects enable customers to move large files at fast speeds over the internet (Wood said he’s seen consistent speeds of 700 megabits per second). This is also why AWS has eliminated data-transfer fees for inbound data, has turned on parallel uploads for large files and created its Direct Connect program with data center operators that provide dedicated connections to AWS facilities.

And if datasets are too large for all those methods, customers can just send AWS their physical disks. “We definitely receive hard drives,” Wood said.

Collaboration is the future

Once data makes its way to the cloud, it opens up entirely new methods of collaboration where researchers or even entire industries can access and work together on shared datasets too big to move around. “This sort of data space is something that’s becoming common in fields where there are very large datasets,” Wood said, citing as an example the 1000 Genomes project dataset that AWS houses.

DNAnexus’s cloud-based architecture

As we’ve covered recently, the genetics space is drooling over the promise of cloud computing. The 1000 Genomes database is only 200TB, Wood explained, but very few project leads could get the budget to store that much data and make it accessible to their peers, much less the computation power required to process it. And even in fields such as pharmaceuticals, Amazon CTO Werner Vogels told me during an earlier interview, companies are using the cloud to collaborate on certain datasets so companies don’t have to spend time and money reinventing the wheel.

No more supercomputers?

Wood seemed very impressed with the work that AWS’s high-performance computing customers have been doing on the platform — work that previously would have been done on supercomputers or other physical systems. Thanks to AWS partner Cycle Computing, he noted, the Morgridge Institute at the University of Wisconsin was able to perform 116 years worth of computing in just one week. In the past, access to that kind of power would have required waiting in line until resources opened up on a supercomputer somewhere.

The collaborative efforts Wood discussed certainly facilitate this type of extreme computation, as does AWS’s continuous efforts to beef up its instances with more and more power. Whatever users might need, from the new 250GB RAM on-demand instances to GPU-powered Cluster Compute Instances, Wood said AWS will try to provide it. Because cost sometimes matters, AWS has opened Cluster Compute Instances and Elastic MapReduce to its spot market for buying capacity on the cheap.

But whatever data-intensive workloads organizations want to run, many will always look to the cloud now. Because cloud computing and big data — Hadoop, especially — have come of age roughly in parallel with each other, Wood hypothesized, they often go hand-in-hand in people’s minds.

Feature image courtesy of Shutterstock user winui.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• It’s time for cloud security and big data to come together
• Buying into big data appliances
• 9 Companies that Pushed the Infrastructure Discussion in 2010

While it seems that chip industry has changed a lot in the last five years or so, that’s nothing compared to what will have to happen going forward. In the last five years the advent of iPhone — which brought real computing on a small, mobile form factor — has upended Intel’s dominance in the PC market. Just last week mobile chip maker Qualcomm surpassed Intel in market cap.

And on the server side and in high performance computing, the demand for more performance in a more efficient package has led to the adoption of new accelerator chips. The Top 500 list of the world’s fastest supercomputers came out Monday and showcases this trend with 62 supercomputers on the list using an accelerator. The fastest computer, called Titan, contains a mix of x86 chips and graphics processors to reach speeds of 17.59 petaflops. It was only four years ago that a GPU-powered computer had even made this list.

The Intel Xeon Phi coprocessor

Already the mobile industry uses a mix of cores to divide different processing jobs on a mobile device as a way to send the right job to the right processor, but also to save battery life by turning cores off when not in use and using a lower power core if possible for a job.

But accelerators can only take the chip industry so far, and are aimed at the high performance computing market. A multipetaflop machine still consumes a lot of power. One way to reduce that power consumption and boost performance is to cram more transistors on a chip by shrinking the space between transistors and reducing the overall size of the chip. This is called moving down the process node, but as the chip industry approaches feature widths of 20 and 14 nanometers, manufacturers are resorting to ever more complicated structures and materials to ensure that the chips perform well.

We’ve covered Intel’s 3-D transistors, Soitec’s new wafers, new materials and a variety of other breakthroughs all designed to keep this process shrinking. But researchers at McGill University have demonstrated that once you get to the atomic scale at levels of sub-five nanometers, the electric current on the chip gets all wonky — delivering a fourfold decrease in current. The researchers theorize when they shrink the channel that the electrons pass through they introduce “quantum weirdness.”

The “quantum weirdness” changes the way the current flows through a channel and so far those changes behave in unpredictable ways because the particles researchers are working with are at the atomic or sub-atomic scale. Which relates the research the top-level chip guys are doing to quantum theory and Schrodinger’s Cat. While Schrodinger’s famous thought experiement is an effort to articulate that the act of observing a quantum particle may change its state, the thought experiment is based on the fact that at subatomic scale materials behave differently.

From a write-up on the research:

As feature sizes in future chips shrink to the level of atoms, the resistance to current no longer increases at a consistent rate as devices shrink; instead the resistance “jumps around,” displaying the counterintuitive effects of quantum mechanics, says McGill Physics professor Peter Grütter.

This means that as chipmakers develop smaller processors or build any electronics with nano materials, the laws of physics will demand they change their materials and likely the structure of those materials, as well as conduct basic research to understand the new rules of playing at the atomic scale. Fortunately scientists at universities and researchers inside labs at place like IBM and HP are thinking about quantum physics and how it will change the rules of semiconductor manufacturing. And yes, there are some that are skipping the electronics part of the equation and skipping straight ahead to quantum computers, but I think our digital bits are here to stay for a few more decades.

Schrodinger’s cat image courtesy of Wikipedia.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Supercomputers and the Search for the Exascale Grail
• Listening platforms: finding the value in social media data
• The Internet of things: creating tomorrow’s health care

It’s a noble goal, but also a necessary one. Big data does have the potential to change our lives, but to get there it’s going to take more than startups created to feed us better advertisements.

Consumer data is easy to get, and profitable

It’s not fair to call the current state of big data problematic, but it is largely focused on profit-centric technologies and techniques. That’s because as companies — especially those in the web world — realized the value they could derive from advanced data analytics, they began investing huge amounts of money in developing cutting-edge techniques for doing so. For the first time in a long time, industry is now leading the academic and scientific research communities when it comes to technological advances.

As Brenda Dietrich, IBM Fellow and vice president for business analytics for IBM Software (and former VP of IBM’s mathematical sciences division), explained to me, universities are still doing good research, but students are leaving to work at companies like Google and Facebook as soon as their graduate or Ph.D. studies are complete, often times beforehand. Research begun in universities is continued in commercial settings, generally with commercial interests guiding its direction.

And this commercial focus isn’t ideal for everyone. For example, Sultan Meghji, vice president of product strategy at Appistry, told me that many of his company’s government- and intelligence-sector customers aren’t getting what they expected out of Hadoop, and they’re looking for alternative platforms. Hadoop might well be the platform of choice for large web and commercial applications — indeed, it’s where most of those companies’ big data investments are going — but it has its limitations.

Enter federal dollars for big data

However, as John Holdren, assistant to the president and director of White House Office of Science and Technology Policy, noted during a White House press conference on Thursday afternoon, the Obama administration realized several months ago that it was seriously under-investing in big data as a strategic differentiator for the United States. He was followed by leaders from six government agencies explaining how they intend to invest their considerable resources to remedy this under-investment. That means everything from the Department of Defense, DARPA and the Department of Energy developing new techniques for storage and management, to the U.S. Geological Survey and the National Science Foundation using big data to change the way we research everything from climate science to educational techniques.

How’s it going to do all this, apart from agencies simply ramping up their own efforts? Doling out money to researchers. As Zach Lemnios, Assistant Secretary of Defense for Research & Engineering for the Department of Defense, put it, “We need your ideas.”

IBM’s Deitrich thinks increased availability of government grants can play a major role in keeping researchers in academic and scientific settings rather than bolting for big companies and big paychecks. Grants can help steer research away from targeted advertising and toward areas that will “be good … for mankind at large,” she said.

The 1,000 Genomes Project data is now freely available to researchers on Amazon's cloud.

Additionally, she said, academic researchers have been somewhat limited in what they can do because they haven’t always had easy access to meaningful data sets. With the government now pushing to open its own data sets, and as well as for collaborative research among different scientific disciplines, she thinks there’s a real opportunity for researchers to do conduct better experiments.

During the press conference, Department of Energy Office of Science Director William Brinkman expressed his agency’s need for better personnel to program its fleet of supercomputers. “Our challenge is not high-performance computing,” he said, “it’s high-performance people.” As my colleague Stacey Higginbotham has noted in the past, the ranks of Silicon Valley companies are deep with people who might be able to bring their parallel-programming prowess to supercomputing centers if the right incentives were in place.

Self-learning systems, a storage revolution and a cure for cancer?

As anyone who follows the history of technology knows, government agencies have been responsible for a large percentage of innovation over the past half century, taking credit for no less than the Internet itself. “You can track every interesting technology in the last 25 years to government spending over the past 50 years,” Appistry’s Meghji said.

Now, the government wants to turn its brainpower and money to big data. As part of its new, roughly $100-million XDATA program, DARPA Deputy Director Kaigham “Ken” Gabriel said his agency “seek[s] the equivalent of radar and overhead imagery for big data” so it can locate a single byte among an ocean of data. The DOE’s Brinkman talked about the importance of being able to store and visualize the staggering amounts of data generated daily by supercomputers, or by the second from CERN’s Large Hadron Collider.

IBM’s Dietrich also has an idea for how DARPA and the DOE might spend their big data allocations. “When one is doing certain types of analytics,” she explained, “you’re not looking at single threads of data, you tend to be pulling in multiple threads.” This makes previous storage technologies designed to make the most-accessed data the easiest to access somewhat obsolete. Instead, she said, researchers should be looking into how to store data in a manner that takes into account the other data sets typically accessed and analyzed along with any given set. “To my knowledge,” she said, “no one is looking seriously at that.”

Not surprisingly given his company’s large focus on genetic analysis, Appistry’s Meghji is particularly excited about the government promising more money and resources in that field. For one, he said, the Chinese government’s Beijing Genomics Institute probably accounts for anywhere between 25 and 50 percent of the genetics innovation right now,  and “to see the U.S. compete directly with the Chinese government is very gratifying.”

But he’s also excited about the possibility of seeing big data turned to areas in genetics other than cancer research — which is presently a very popular pastime — and generally toward advances in real-time data processing. He said the DoD and intelligence agencies are typically two to four years ahead of the rest of the world in terms of big data, and increased spending across government and science will help everyone else catch up. “It’s all about not just reacting to things you see,” he said, “but being proactive.”

Indeed, the DoD has some seriously ambitious plans in place. Assistant Secretary Lemnios explained during the press conference how previous defense research has led to technologies such as IBM’s Watson system and Apple’s Siri that are becoming part of our everyday lives. Its latest quest: utilize big data techniques to create autonomous systems that can adapt to and act on new data inputs in real time, but that know enough to know when they need to invite human input on decision-making. Scary, but cool.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011

Generally speaking, we’ve already seen how crowdsourcing can be an effective method for solving big data problems. The Netflix Prize challenge in 2009 attracted more than 50,000 participants trying to improve Netflix’s Cinematch algorithm, and today we have Kaggle — an entire company dedicated to hosting competitions for companies trying to crowdsource their own analytical challenges. And it’s the cloud, with its centralized nature, virtually unlimited and on-demand resources, that makes it possible to have so many people access and work with the same data sets at the same time.

It’s true, of course, that big data doesn’t necessarily connote scientific workloads, but scientific workloads do increasingly rely on big data techniques. Some refer to data as the fourth paradigm of science because the sheer amount of data available and the new technologies and techniques for working with it are fundamentally changing how scientists go about their research. This has been going on for quite a while, actually, hence the massive research networks connecting supercomputers and research centers across the world. Researchers needed a way to transfer massive data sets to their peers to run on their systems, so they built networks such as the National LambdaRail, XSEDE and CERN’s Large Hadron Collider network.

However, while this arrangement might work fine for researchers working on projects for national labs or universities, who also happen to have time reserved on supercomputing systems, it’s not entirely democratic. Enter cloud computing. Now, anyone can have access to supercomputer-like processing power and, equally important, centralized data sets that don’t require a 40 Gbps connection to download. Companies such as DNAnexus rely on the cloud to host massive genomic data sets on which scientists can collaborate, and also to power those scientists’ computations on the data.

And although companies such as DNAnexus focus more on collaboration than on crowdsourcing, the tools for crowdsourcing are in place. Today, for example, I read about a company, Life Technologies, which makes semiconductor chips that actually carry out a variety of genome-sequencing workloads. Life is hosting a competition within its online community to improve the speed, scalability and accuracy of chips. Contestants will have access to the raw data as well as cloud-based resources for running computations.

Critics can call cloud computing overblown until they’re blue in the face — they might even be right when it comes to certain business applications — but there’s no denying the effects it could have in the scientific world. By giving virtually anybody access to relevant scientific data sets and the resources necessary to analyze them in a timely manner, cloud computing could result in real answers to some previously perplexing questions.

Feature image courtesy of Flickr user Kennisland.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• 12 tech leaders’ resolutions for 2012
• Finding the Value in Social Media Data

The availability of cheaper yet powerful computing helps scientists crunch massive amounts of data that can lead to new discoveries. Last month, I spoke to a researcher at Johns Hopkins about this. Dr. Alex Szalay, of Johns Hopkins said:

“In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st-century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay said.

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing which will lead to new advances in astronomy and physics, much like the microscope’s creation in the 17th century led to advances in biology and chemistry.

This is why the search for exascale supercomputing matters. But compute in isolation isn’t enough. The other elements that are coming together are faster networks to move data around (although they aren’t fast enough) and a more collaborative culture in science and technology across geographies and disciplines. Broadband helps with that of course. The use of open-source software in scientific computing and of cheaper ways to harness massive compute power, either though the cloud or through GPUs, also allow more people to play with machines that are akin to the top-of-the-line supercomputers of 5-6 years ago.

So the democratization of compute, data analytics, the data itself and fast networks are changing how deeply scientists can look at a problem. And it offers a wider lens to put research in context thanks to collaboration and interdisciplinary studies. Today it’s a new form of chemical bonding, but next it could be a revolution in energy production inspired by the new bond. And with these new tools, discoveries will happen faster and be applied in more places. That’s cool even if chemistry isn’t your thing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Finding the Value in Social Media Data
• 9 Companies that Pushed the Infrastructure Discussion in 2010

Servers? We don't need no stinkin' servers!

Both mobile and high-performance computing are placing huge power efficiency and performance demands on chips, but the $64,000-question is how long until such extreme computing use cases hit the server mainstream. Asked another way, the question becomes, how long until Amazon Web Services adopts ARM-based servers?

Or perhaps it isn’t ARM-based servers, but a variation on an Intel chip that takes its architecture from some of the more innovative and energy-efficient silicon out today. For example, Adapteva, a startup I profiled back in May, released a 64-core chip on Monday that can deliver 70 gigaflops of performance per watt. If you don’t speak gigaflops, that’s okay: It basically has done what Intel and certain countries have deemed impossible with the current generation of silicon.

The government of the European Union, in its quest for an exascale supercomputer, has targeted a goal of getting 50 gigaflops per watt (Intel also thinks this would work). In conversations with folks that design supercomputers, the thinking is that a conventional x86-based machine would require the equivalent of a power plant or two to run. That includes all the networking and other trimmings, but the bottom line is that Adapteva’s chips deliver more flops per watt, and that’s a good thing.

It’s not just supercomputers though. Adapteva’s CEO Andreas Olofsson told me the company is only targeting computing extremes such as supercomputing and mobile phones because that’s where the power efficiency pain point is today. Because mobile phones run on batteries, and no one wants a smartphone that dies after two hours, vendors using ARM’s power-efficient architecture have dominated the mobile sector. When Microsoft adapted Windows to run on ARM, it spoke volumes about the need for power efficiency. Windows is one of the most x86-oriented pieces of software out there.

These shifts in usage profiles and the high demand for compute are creating opportunities for companies like Adapteva, so it’s not too far-fetched to wonder how long until that pain point hits conventional servers.

I often cover companies that are hoping the combination of monolithic applications and a desire to reduce power consumption means webscale and cloud vendors will embrace a new architecture. Companies such as Tilera, SeaMicro, Adapteva, Calxeda and others are all betting the next gear Facebook or Amazon buys will be their hardware or contain their chips.

However, even in its state-of-the-art data center optimized at the server level to be energy-efficient, Facebook challenged the way servers and data centers are built but didn’t touch the silicon itself. So, clearly, the webscale world isn’t champing at the bit to replace the x86-based servers their applications are running on. SeaMicro even has shown charts demonstrating that the CPU is only a third of the power associated with running a server, which means there’s still plenty of fat to trim. Of course, SeaMicro is building a server that trims that non-CPU fat and runs Intel’s Atom chips.

However, the global demand for energy and the supply we currently have are reaching a point where it’s safe to conclude that power consumption will become a greater cost and constraint associated with operating data centers. And at some point, building in cooler climates, hot and cold aisle containment, and even newly designed servers won’t be enough if the silicon itself is too hot.

So the question isn’t if, but when, server companies abandon the PC-style architecture. Perhaps Intel, AMD or Via will continue to tweak x86 silicon until it can perform more calculations using less power, or perhaps it will be time for Amazon or Microsoft Azure to go with ARM, Tilera — or even Adapteva.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cleantech
• How tomorrow’s mobile-centric data centers will look
• Cloud computing’s impact on chip and hardware design

The concept of an accelerator is a familiar one in super computing, where the addition of a specialized massively multicore chip, such as a graphics processor or custom chip, is becoming more common. But unlike a GPU, the 64-core Adapteva chip only operates at one watt. To understand how powerful that is from an energy efficiency perspective, a four-to-eight-core server chip could operate at anywhere from 60-120 watts. And the challenge of building the next generation of supercomputers is constrained by the power demands such massive supercomputers would require.

But Andreas Olofsson, the founder and CEO of Adapteva, isn’t focusing on the HPC market at first–despite asserting that his design can scale to a 4,096-core design that would run at 64 watts. He said that while there is plenty of talk about low-power computing, “As long as you can plug something into a wall, the need for low power goes down significantly. It’s only a little bit painful.” However, in the mobile world where devices need to run all day, yet avoid bulky batteries, power consumption is at a premium. So that’s where Adapteva will focus for its big push (the company has some military applications as well).

The company began in 2008 and has managed to raise $2 million in funding from angels and boardmaker BittWare, its first customer. Amazingly, with that small amount of funding it has managed to have three versions of its chip built, making the startup incredibly capital efficient. However, the goal isn’t to build chips for the mobile market, but to license the technology, much like ARM, the firm behind the most common architecture in mobile phones, does. BittWare will manufacture the Adaptevea chip design — called the Epiphany– on its boards.

But in a highly competitive market, and especially on smarthphones, where space on the board is at such a premium, will device makers really embrace an unproven and as-yet-unneeded chip? Olofsson has two more difficult tasks to accomplish (since he’s apparently taken care of the hard task of building and designed a 64-core chip that runs at 1 watt for less than $2 million.) He must explain to board makers, chip firms and device makers why gadgets need this rather foreign accelerator chip, and he has to convince them that it makes sense to process data on the phone, rather than ship it over the cellular network.

The first task is made easier by the low-power envelope and by the fact that the full 64-core system on a system is fairly small — about 8mm square Olofsson said. Check out the model of the A5 system on a chip used inside Apple devices provided below to see how much space the Epiphany chip can take up. It would have to replace existing GPUs in this case.

The second task may be made easier by people’s desire to handle tasks such as speech or facial recognition or intense video games on their mobile devices. Olofsson argues that the latency inherent in sending even voice recognition to a server is problematic and that gameplay is impossible. Plus it costs more in terms of data charges and can drain the battery. “If you can keep the radio quiet and use the processing locally the battery life gets better,” he said.

Like many visionaries pushing a new technology he’s not entirely sure how the Epiphany could change mobile computing, but he’s certain that by boosting performance on smartphones to this degree it will. I’m eager to see if mobile device makers agree.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What the Google-Motorola deal means for Android, Microsoft and the mobile industry
• Bluetooth to Feel Blue as Personal Area Network Battles Loom
• HTML5’s a Game-Changer for Web Apps

Dr. Aditya Vailaya, chief scientist of consumer-electronics recommendation site Retrevo, doesn’t discount the Watson victory, but he does recognize the clear line between machine learning (the broad umbrella under which Watson’s question-answering algorithms reside) and artificial intelligence. That a system such as Watson can understand natural language is a huge step forward in machine learning, he explained, but it’s still only as good as the data it has to work with and the algorithms, or rules, developed to process that data. Put more simply, machine learning is a largely a case of “garbage in, garbage out.”

Vailaya’s comparison of Watson and IBM’s formerly most-famous human destroyer, Deep Blue, is illustrative of this point. At their cores, he said, both systems just do math: They use computer processors to analyze every possible outcome or answer, whereas humans use instinct to eliminate some possibilities and narrow their focus. Deep Blue analyzed a chess board, its possible moves and the possible outcomes of each possible move — and it went many, many levels deep. Watson essentially does the same thing, he explained, only in a less-constrained environment where nothing is as finite as chess moves. In processing natural language, Watson needs to decipher key words and their context, then analyze that information against its enormous data set.

What makes natural-language processing so difficult, though, is making sense of poorly written language, where it can be difficult even for humans to decipher subjects, verbs and other parts of speech. Watson’s ability to do this is based wholly on its human “trainers’” understanding of language and the algorithms with which they are able to program Watson based on this understanding. The same will hold true when, as has been announced, IBM begins working to implement Watson in the health care field. Vailaya explained that not only will Watson need to be loaded with a vast amount of medical data, but it will need very knowledgeable medical experts and computer scientists to develop algorithms that enable Watson to analyze symptoms against that data. If Watson were crawling the web to bolster its data store, humans would have to develop rules to help it decide which sources are providing correct, usable data, something Vailaya already does with Retrevo.

This is in contrast to artificial intelligence, where machines are designed to mimic human and/or animal brains and determine their own rules about what to do in any given situation. Vailaya describes one process of creating artificial intelligence as giving machines “emotions” and implementing some sort of incentive system. Such a machine would then remain in its environment and create its own rules for making decisions based on past experiences. In essence, Vailaya explained, the machine would be self-motivated to act in its own best interests. Although there are some small-scale artificial-intelligence implementations in areas like gaming and robotics, he noted it’s a very complex field, and even still, scientists are limited to building only what they can understand. If artificial intelligence were as far along — or as easy (relatively speaking) — as machine learning, then humans might have something to worry about.

Don’t get Vailaya wrong, though. He acknowledges it’s a big deal that IBM and its research partners were able to develop a computer that can answer natural-language questions better than humans can. Because of computers’ inherent ability to learn spaces more deeply than humans can, systems like Watson have the potential to be very useful “in any area where [a professional] need[s] an expert consultant” to answer difficult questions in a hurry or, ultimately at the consumer level. From Vailaya’s perspective, though, we’re a long way off from having to “welcome our new computer overlords.”

For many deeper discussions about data-analysis algorithms, attend our upcoming Structure Big Data conference March 23 in New York City, where data scientists will discuss the algorithms in place for analyzing everything from Twitter streams to market data.

Related content from GigaOM Pro (sub req’d):

• Cloud Computing Reaches the Final Frontier
• Supercomputers and the Search for the Exascale Grail
• We Can Call It a Cloud, But It’s Still Hardware

Superfast SSDs Are Coming, but Will They Be Used the Right Way? (From Ars Technica) Many cutting-edge enterprise storage and database appliances already utilize SSDs in just this manner, and some even have algorithms to determine what is stored on them.

Will the Cloud Mean Joblessness for You? (From The Register) The great cloud-computing-will-destroy-jobs debate continues. I think the author is on the right track, but might paint too rosy a picture. After all, new jobs might not fit present employees.

I don’t want to go into what the spread of unboxing videos may mean for our culture, but I do want to point out the most random and incredibly sincere unboxing video I’ve come across — that of high-performance computing gear from IBM, which is acting as the warm-up system for the coming Blue Waters petaflop supercomputer at the National Petascale Computing Facility at the University of Illinois-Urbana Champaign (hat tip Inside HPC). The video isn’t a play-by-play unboxing because that would take too long, but is a lovingly shot homage to their new gear that I figured fellow geeks might appreciate.

Related GigaOM Pro Content (sub req’d): Supercomputers and the Search for the Exascale Grail