As anyone even casually familiar with parallel processing knows, running applications across more nodes means jobs execute faster because they’re able to share the computing workload. The more cores, the faster it runs. This what makes Hadoop, for example, so great at processing large chunks of data. The MapReduce framework on which it’s based divvies up the work across nodes and everything they find is stitched back together as the result of a job.

But even Hadoop can only scale to tens of thousands of nodes and, because of its focus on “nodes,” actually isn’t really good at utilizing multi-core processors to their fullest (expect to hear more about the limitations of Hadoop at our Structure: Data conference March 20-21 in New York). The IBM-built Sequoia supercomputer (housed at Lawrence Livermore National Laboratory) that the Stanford team used consists of 98,304 processors (or nodes), each containing 16 computing cores. That’s a grand total of 1,572,864 cores, and the researchers were able to use the majority of them, which they claim is a record of some sort.

Sequoia, decomposed

But record or not, that’s an incredibly complex undertaking. Programming the jet-engine simulation meant figuring out how to divvy the code into more than a million different tasks that could run across tens of thousands of nodes and 16 cores within each of those nodes. If even one of those processes is buggy, it could slow down or ruin the whole simulation.

Even in the world of supercomputing, where systems now regularly contain hundreds of thousands of cores — some of them special-purpose GPU co-processors — there’s a shortage of programming talent to actually use them all to their fullest potential. As my colleague Stacey Higginbotham explained in some time ago, the world of high-performance computing is hurtling toward exascale computing but a bigger problem than energy-consumption might be finding applications that need that much computing power and the algorithms capable of operating at that scale.

Still, the implications of advances in parallel programming are huge — like potentially life-altering huge. This is true not only because of the scientific questions we’ll soon be able to answer at speeds inconceivable even a decade ago, but also because of the computing power we’ll all soon be carrying around in our pockets and purses. If you think those multi-core smartphones and tablets are great now because they can run multiple applications at the same time, just wait until their processors are even bigger and badder and we have more applications — photo- and video-editing, computer-aided design, games and who knows what else — that can actually get the most out of them.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Dissecting the data: 5 issues for our digital future
• Real-­time query for Hadoop democratizes access to big data analytics

Reality disappointed: What we actually got was a limited library of not-new games (Homefront, anyone?), many of which you already owned, but even laggier than on your own hardware. Turns out traditional retail, game publisher, and hardware platform companies made it difficult for cloud gaming services to get the best games on the day of release, and even then the gameplay quality was slightly inferior.

But the concept of gaming in the cloud is still an idea worth pursing for a far greater promise: the ability to deliver an entirely new kind of game experience.

Historically, in games as in any other media, new distribution technologies enable new creative experiences. Pong wouldn’t have been possible without a new device plugged in to your TV. Internet-connected computers meant you could play Duke Nukem and Quake with other people online. The evolution of server technology brought massively-multiplayer games. The iPhone brought Angry Birds, a game designed for a touch interface, and so forth.

So why should a cloud gaming service be used to deliver the same old games as before that were built for a $250 machine?

What we should be wondering, then, is what new kinds of games and gaming experiences cloud delivery could inspire? Compared to the gaming hardware you own, a cloud gaming service could access much more computing power—with a limitless capacity to add processing. Consider after all that the most powerful supercomputer in the world, the Titan, is about 70,000 times more powerful than an Xbox 360. Granted the Titan costs a cool $100 million, which cuts out most households, but scaling back to basic and accessible data center prices would still offer many orders of magnitude more computing power than any current or near-future home console . (And this isn’t to say great gaming experiences are limited to powerful hardware—to the contrary mobile phones play compelling games, too. They’re just of a different sort.)

As for content itself, games purpose-built for the cloud do not yet exist — ones that aren’t encumbered by the limits of processing power, that would use the full advantage of many more, and more powerful, CPUs and GPUs. These “supercomputer games” would open up creative possibilities far beyond what games of today are capable.

Imagine supercomputer games with vividly lifelike worlds and characters (and not the almost-real, uncanny valley of current-generation graphics), or a single battlefield with 50,000 other players playing at the same time — or opponent AI on the level of IBM’s Jeopardy!-winning Watson. Supercomputer games could be dramatically different from anything you can play tonight at home. I’m no game designer, but what if we could use real-time traffic data to fill the streets of the next Grand Theft Auto, or step into a computer-generated world that looks as compelling as the Lord of the Rings movies?

Now, there are many reasons, beyond the technological, that these games don’t yet exist: It would be prohibitively expensive to pay artists to create all those detailed graphics, and simple AI is good enough to defeat most any player at most any game. But the record of creative innovators is that eventually they find a way to stretch the available technology to its limit. And some gamemakers are already beginning to probe at the games you can create if you host some of the game in the cloud.

There is a nagging constraint to the cloud, of course — bandwidth, which simply isn’t growing at the pace of Moore’s law. Network latency makes fast-twitch games, in which defeat is determined in microseconds (like with the top console genre, first-person shooters) hard to play over today’s internet. So, at least until the next engineering breakthrough, these supercomputer games might be designed around genres requiring slower player reflexes than, say, Call of Duty or StarCraft.

Best of all, the only hardware you would need at home is a basic input device like a controller and a box to render the graphics, and it could be cross-platform so that you could play from a PC or Mac or any smartphone. As one for-instance, OUYA, the new open, Android-based console I back, could be great for a cloud-delivered game (hear me, developers?), and its notable that Sony bought up Gaikai and certainly has plans.  (Full disclosure:  OUYA also has an announced partnership with the relaunched OnLive.)

Supercomputer games could be extraordinary. Now some intrepid game developers just have to make one.

Roy Bahat is Chairman of the open, Android-based game console company OUYA, and is former president of IGN. He is also on the faculty at UC Berkeley. Follow him on Twitter @roybahat

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What Enterprise Software Vendors Could Learn from the Consumer Space
• Virtual Worlds: Trends and Opportunities
• GigaOM Research highs and lows from CES 2013

If there’s a report detailing Leetaru’s findings, I haven’t been able to find it, but even this snippet from the project’s Facebook page is pretty insightful:

From this analysis, Wikipedia is seen to have four periods of growth in its historical coverage: 1001-1500 (Middle Ages), 1501-1729 (Early Modern Period), 1730-2003 (Age of Enlightenment), 2004-2011 (Wikipedia Era) and its continued growth appears to be focused on enhancing its coverage of historical events, rather than increased documenting of the present. The average tone of Wikipedia’s coverage of each year closely matches major global events, with the most negative period in the last 1,000 years being the American Civil War, followed by World War II. The analysis also shows that the “copyright gap” that blanks out most of the twentieth century in digitized print collections is not a problem with Wikipedia where there is steady exponential growth in it’s coverage from 1924 to today.

Leetaru also visualized the findings and created a couple of 30-second videos showing Wikipedia coverage and sentiment over time and geopgraphy. His visualizations (like the one above mapping “[e]very year from 1000 AD to 2012 referenced in Wikipedia plotted and cross referenced when mentioned in the same article”) are beautiful as works of art, although one can’t readily decipher who the influential people, organizations and years are.

Still, the project is a valuable reminder of just how far we’ve come in terms of data-analysis techniques and the computing power necessary to run them. This is why the idea of big data is so popular, even if the possibilities haven’t been fully realized yet. Analyses that would have taken weeks or days now take hours, minutes or seconds, which means anyone with the right data and the right gear can learn a heck of a lot if they can keeping coming up with good questions.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Finding the Value in Social Media Data
• The importance of putting the U and I in visualization

There are three problems with building a telescope capable of reading radio waves from that far out in deep space (actually there’s a real estate problem too, because the array will require millions antennas spread over an area the width of the continental U.S., but we’ll stick to computing problems). The first problem is the data that this Square Kilometre Array (SKA) will generate. IBM estimates it will produce:

… a few Exabytes of data per day for a single beam per one square kilometer. After processing this data the expectation is that per year between 300 and 1,500 Petabytes of data need to be stored. In comparison, the approximately 15 Petabytes produced by the Large Hadron Collider at CERN per year of operation is approximately 10 to 100 times less than the envisioned capacity of SKA.

And guys, the LHC is in the midst of getting its own cloud computing infrastructure in order to handle its data. So this IBM/ASTRON project may be just the beginning for SKA. As I say in the headline, in many ways, projects like the LHC and the SKA are ambitious investigations into the origins and composition of the universe. Our investigations into dark matter will require a compute effort that could rival the engineering effort that it took to get men on the moon. Which makes big data our Sputnik and our Apollo 11.

Now, back to the problems associated with the telescope. It will generate data like a corpse breeds maggots, so the project needs a computer big enough to process it without requiring a power plant or two. Additionally that data might have to travel from the antenna arrays to the computer, which means the third problem is the network. I’ve covered the need for compute and networks to handle our scientific data before in a story on Johns Hopkins’ new 100 gigabit on-campus network, but the scale of the DOME project dwarfs anything Johns Hopkins is currently working on. From that story:

[Dr. Alex Szalay of Johns Hopkins] ascribes this massive amount of data to the emergence of cheap compute, better imaging and more information, and calls it a new way of doing science. “In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st-century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay said.

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing which will lead to new advances in astronomy and physics, much like the microscope’s creation in the 17th century led to advances in biology and chemistry.

So we need the computing and networking equivalent of a microscope to enable us to deal with a telescope planned for 2024, and the time to start building it is now. That gives us a lot longer than the time frame we had to land on the moon. IBM views the problem as one worthy of the following infographic:

As the infographic shows, we’re going to need massively multicore, low-power computers, better interconnection using photonics and new ways of building our networks. Hopefully, the search for dark matter is worth it.

SKA image courtesy of the Square Kilometer Array.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future

(c) 2012 Pinar Ozger. pinar@pinarozger.com

Speaking at GigaOM’s Structure:Data conference, Los Alamos HPC deputy division leader Gary Grider said that the exascale computer has so many parts, that some element will constantly be failing.

“It wouldn’t be worth building if it didn’t stay working for more than a minute,” Grider said. “Resilience is absolutely a must. The way you get answers to science is you run problems on these things for six months or more. If the machine is going to die every few minutes, that’s going to be tough sledding. We’ve got to figure out how to deal with resilience in a pretty fundamental way between now and then.”

Grider and Los Alamos’s technology partners have between 6 and 10 years to work on the problem, and the national lab won’t be alone. According to inside-Data president Rich Brueckner, who moderated the “Faster Memory, Faster Compute” panel Grider spoke on, countries from all over the world are in an exascale race. Brueckner said it’s just as likely as Russia, Japan, China, India or the European Union develops the exascale machine as the U.S.

Watch the livestream of Structure:Data here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

Cray's XK6 supercomputer

It looks like Oracle has some competition when it comes to selling big iron for big data. On Wednesday, Cray, the Seattle-based company best known for building some of the world’s fastest supercomputers, said it’s getting into the big data game. A new division within Cray, called YarcData, will leverage Cray’s experience working within data-intensive environments for customers such as Boeing in order to woo large-enterprises with big data needs.

Cray was short on details in a press release announcing the new division, but new YarcData SVP and GM Arvind Parthasarathi, formerly of Informatica is quoted saying, “YarcData is the nexus of the world’s most advanced technologies from Cray being applied to solve the world’s most challenging Big Data problems.” The natural leap is that Cray will design parallel-processing systems capable of incredible data throughput — something already required in the supercomputing space, where incredible processing capacity would be wasted without a steady data stream — but that will support today’s popular big data tools (e.g., Hadoop, analytic databases and predictive analytics software).

This type of system could be very valuable for organizations such as banks and intelligence agencies that want to run big data workloads as fast as possible — even process streaming data in real time– and the deep pockets to pay for Cray’s presumably pricey systems. Despite the fact that big-data framework Hadoop gained popularity in part because it’s designed to run on commodity hardware, there’s always a place for high-end hardware when milliseconds really do matter, and there’s something to be said for pre-configured systems that take the guesswork out of building a big data environment, as I explained recently in a piece for GigaOM Pro (sub req’d).

Cray isn’t alone in pushing this high-performance, enterprise-focused big data vision, though. Oracle made a splash in October when it announced a Big Data Appliance that marries Hadoop, R, NoSQL and other technologies to the high-end hardware Oracle obtained when it bought Sun Microsystems. IBM also has an extensive big data software portfolio complemented by a systems business that includes supercomputers, as well. And although it doesn’t have an HPC pedigree like the others, Teradata has years of experience building systems optimized for analytics.

Cray won’t likely become a household name in the big data world, and its notoriously secretive customers might never divulge what they’re using its analytics products for, but there certainly is a market — however small — for super-big, super-fast and super-expensive data.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Buying into big data appliances
• The importance of putting the U and I in visualization
• A near-term outlook for big data

The availability of cheaper yet powerful computing helps scientists crunch massive amounts of data that can lead to new discoveries. Last month, I spoke to a researcher at Johns Hopkins about this. Dr. Alex Szalay, of Johns Hopkins said:

“In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st-century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay said.

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing which will lead to new advances in astronomy and physics, much like the microscope’s creation in the 17th century led to advances in biology and chemistry.

This is why the search for exascale supercomputing matters. But compute in isolation isn’t enough. The other elements that are coming together are faster networks to move data around (although they aren’t fast enough) and a more collaborative culture in science and technology across geographies and disciplines. Broadband helps with that of course. The use of open-source software in scientific computing and of cheaper ways to harness massive compute power, either though the cloud or through GPUs, also allow more people to play with machines that are akin to the top-of-the-line supercomputers of 5-6 years ago.

So the democratization of compute, data analytics, the data itself and fast networks are changing how deeply scientists can look at a problem. And it offers a wider lens to put research in context thanks to collaboration and interdisciplinary studies. Today it’s a new form of chemical bonding, but next it could be a revolution in energy production inspired by the new bond. And with these new tools, discoveries will happen faster and be applied in more places. That’s cool even if chemistry isn’t your thing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Finding the Value in Social Media Data
• 9 Companies that Pushed the Infrastructure Discussion in 2010

The original Cray supercomputer

In the past decade supercomputers were dressed-up versions of Intel’s x86 machines, but increasingly supercomputers are borrowing innovations (and silicon in the form of ARM-based chips or DSPs) from the mobile and big data realms to add speed without guzzling too much power.

Prior to this century many supercomputers really were a different animal entirely, sporting specialty chips and software. But the industry turned to commodity chips in the early 2000s. Now, to meet the demands of exascale computing at low power, chip makers are taking inspiration from the cloud computing and mobile industries.

ARM tries supercomputing on for size

As the Supercomputing 2011 show gets under way in Seattle, Nvidia, Texas Instruments, ARM and others are announcing new silicon to power the machines we rely on for science, climate prediction and high-end simulations in industries that range from oil production to car design.

Nvidia is a fairly recent newcomer to the supercomputing market, but it has made huge strides since 2008, when it first starting pushing its graphics processors (GPUs) as a way to boost speed while keeping energy usage in check. It said it would use its high-end GPUs and its new GPU-plus-ARM chip to build a new supercomputer in Spain. This is the first time an ARM-based processor has made its way into a supercomputer. ARM thus far has been the chip of choice inside cell phones and tablets.

Accelerator chips advance in supercomputers

Japan's K supercomputer is the fastest in the world.

Nvidia is doing well with its GPUs, given that in the top 500 ranking of the world’s fastest supercomputers, 39 systems use GPUs as accelerators and 35 of these use Nvidia chips. The graphics processors are used in supercomputers because they can handle massively parallel tasks that high-end computing requires while using less energy than the typical CPUs made by Intel and AMD. Nvidia and its GPUs made their first appearance on the list in 2008, and the last time the top 500 list was published, six months ago, Nvidia chips were in 17 machines. To go to 35 today is a pretty big uptake.

Perhaps inspired by Nvidia’s success in getting its GPUs onto supercomputers, Texas Instruments is bringing its digital signal processors to the mix for high-performance computing. DSP chips are really good at math, and they are used in telecommunications chips and in routers. TI has been thinking about this for a while, but Monday was its first launch into the market formally.

New chips for the cloud

The same power-efficiency issues that plague those trying to advance supercomputing are hitting those who run webscale applications, from Facebook to Amazon Web Services. And while the cloud and web-scale data center operators aren’t looking for specialty gear, like Infiniband for networking, they are running one or a few applications on their hardware, similar in some ways to a supercomputer, where all workloads are optimized for speed.

This is why certain chip and hardware companies, such as Tilera, Calxeda and Applied Micro, see an opportunity to redesign the silicon and gear inside the cloud. Meanwhile, companies such as Adapteva, which makes a massively multicore chip for cell phones and HPC, see an opportunity in pushing into supercomputers and mobile handsets, where the need for more-powerful processors and lower power consumption are always at war. And with ARM piggybacking on this trend thanks to Nvidia, it’s clear that supercomputers want to be super without the influence of PCs.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The fourth quarter of 2012 in cleantech
• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Supercomputers breached the petascale barrier back in 2008 with IBM’s Roadrunner and Cray’s Jaguar performing more than a million billion calculations per second, and thoughts immediately turned to the next obvious milestone, achieving one billion billion calculations a second. But instead of speeding up processors and the networking technology inside the machines, researchers are going to have to think first and foremost about power. Running an exascale supercomputer could require up to two city-sized power plants if scientist build the supercomputer out like the current machines. That’s not going to fly.

And because supercomputers now tend to run more mainstream components and software, the answers researchers find for their power and performance problems may be of use sooner rather than later for webscale computing or corporate IT departments. Perhaps we’ll see an ARM-based supercomputer or an entirely new architecture emerge if this funding is actually allocated. Maybe we’ll get a more power efficient optical networking technology, that could scale more cheaply for use inside the data center (going all optical saves power because it eliminates the need to switch a fiber optical signal back to an electrical one). The President’s budget may not pass Congress, but these dollars aren’t just about an obscure search for high-end machines, they could benefit and get benefits from more traditional computing innovations.

Related content from GigaOM Pro (subscription req’d):

• Supercomputers and the Search for the Exascale Grail
• Pushing Processors Past Moore’s Law
• Think Converged Infrastructure Means Lock In? Think Again.

IBM announced eight universities Friday — Massachusetts Institute of Technology, University of Texas, University of Southern California, Rensselaer Polytechnic Institute, University at Albany (NY), University of Trento (Italy), University of Massachusetts, and Carnegie Mellon University — that have contributed to Watson thus far. Their efforts range from MIT’s work on START, an “online natural language question answering system … which has the ability to answer questions with high precision using information from semi-structured and structured information repositories,” to RPI’s work on “a visualization component to visually explain to external audiences the massively parallel analytics skills it takes for the Watson computing system to break down a question and formulate a rapid and accurate response to rival a human brain.” It’s all high technology, though, and it helps Watson figure out where to look for information, how to learn from previous questions and, ultimately, to decide whether it’s confident enough to buzz in.

I got a taste of Watson in August when I toured IBM’s Industry Solutions lab in Hawthorne, N.Y., and I have to say it was impressive. Without giving away any of the secrets behind Watson — or any of its potential weaknesses — Principal Investigator for DeepQA David Ferucci gave a sampling of just how deep Watson goes to determine possible answers then pare them down to a final one. It was impressive, and I wasn’t surprised when Watson came out of January practice around ahead of its human competition. As someone who once drove to Los Angeles to try out for Jeopardy, I appreciate how difficult it is for humans to make these types of judgments, and how difficult it must have been to program a computer to do the same.

The techie in me wants Watson to win so the world gets an understanding of what’s possible with algorithms, even beyond the customized experience of browsing Amazon.com, but the human in me wants to cling to that last thread of hope that human judgment can prevail against artificial intelligence. The realist in me knows that Watson will prevail, though, and AI guru Ray Kurzweil agrees. I guess we can all take solace knowing that it takes humans like those who worked on Watson to write such complex software and build such complex systems — for now, at least.

Image courtesy of IBM.

Related content from GigaOM Pro (sub req’d):

• Cloud Computing Reaches the Final Frontier
• Supercomputers and the Search for the Exascale Grail
• We Can Call It a Cloud, But It’s Still Hardware