GigaOM » networks

How to deliver the next-generation web experience

Tue, 22 Jan 2013 07:55:36 +0000

Delivering a positive web experience has become exceedingly more complex as the access environment has shifted from a desktop-centric vision to one that is increasingly focused on mobile devices. Mobilizing web design is a catch-22; adjusting to design challenges is costly, but not adjusting is equally costly, because a poor mobile web experience results in a loss of revenue. This report will examine what drives content consumption today and illustrate what the changing consumption of content has meant to the development and delivery of web and mobile content. It will also examine the evolution of the web experience and explore the challenges of content delivery to both mobile and desktop devices.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The role of converged infrastructure in the data center

Fri, 28 Dec 2012 13:55:35 +0000

GigaOM Research projects that the cloud computing market will grow from $70.1 billion in 2012 to $158.8 billion in 2014. This adoption comes with a compensatory need for sustainable performance from cloud service providers. However, this increased performance cannot be sustained if the corresponding cost to the service provider (SP) for delivering this performance also increases. What service providers need is a way of delivering low latency, fast response, and increasing performance while minimizing the cost of the network.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why you should be afraid of the UN’s plan to regulate the internet

Mathew Ingram — Mon, 03 Dec 2012 20:20:53 +0000

It’s rare that ordinary internet users need to care about what goes on at the United Nations, but this is definitely one of those times, if only because the UN’s telecom arm is currently holding hearings in Dubai that could change the way the global network functions in some important ways. Although fears of what some have described as a UN “takeover” of the internet are over-blown, some of the proposals the telecom committee will be considering could have ramifications for the way we use the internet, and perhaps more importantly how we pay for it. They are serious enough that Net veterans like Vint Cerf, one of the “fathers of the internet,” are warning of the dangers if these proposals are actually adopted.

As we described in a post earlier this year, the controversy stems from a meeting of the UN’s International Telecommunications Union or ITU — a body that is made up of representatives from all the countries that belong to the United Nations. The core mandate of the group, as its name implies, is to get national governments and regulators to agree on rules that govern the way telecommunications networks operate between different jurisdictions. Now, the ITU wants to extend its oversight to the internet, and it is asking members to vote on new rules that would govern the global computer network.

“Our protocols were designed to make the networks of the Internet non-proprietary and interoperable. They avoided ‘lock-in,’ and allowed for contributions from many sources. This openness is why the Internet creates so much value today. Because it is borderless and belongs to everyone, it has brought unprecedented freedoms to billions of people worldwide.” — Vint Cerf

One of the things that makes this issue so difficult to pin down — which in turn has caused a lot of fear-mongering and hyperbole about the outcome — is that the ITU doesn’t make its meetings public, nor does it release much specific information about the proposals that are going to be considered by the group during its meetings over the next couple of weeks. As a result, most of the information we have comes from leaked documents and second-hand or third-hand reports about the discussions. The internet’s current governing body, the Internet Corporation for Assigned Names and Numbers, has been accused of being secretive in the past, but it is an open book compared to the ITU.

As a number of sources have pointed out, there are a wide range of proposals from dozens of different countries who belong to the UN, and most of them will never see the light of day or become international law. But the proposals that should be of concern to users — and have drawn criticism from Vint Cerf, now a Google evangelist, as well as a multi-party group of U.S. legislators and other agencies — fall into two large buckets:

Limits to free speech and the free flow of information

One of the reasons to be afraid of what the ITU might do is that some UN member states — including Russia, Turkey, Iran and others — would like the power to restrict speech and information, and their proposals to the UN body are aimed at making it easier for them to do that. Obviously, China and other countries have shown that they are capable of creating national-level firewalls and systems that can monitor and block whatever forms of communication they wish, so the ITU’s approval isn’t necessary for this to happen.

If regulations were changed, however, the fear is that it could become even easier for countries like Egypt or Syria to filter and block specific online content, rather than having to use brute force to shut off the internet altogether — something that is much more obvious than a secretive filtering or controlled-access scheme, and therefore easier to criticize or defend against.

“Governments are trying to use a closed-door meeting of The International Telecommunication Union that opens on December 3 in Dubai to further their repressive agendas. Accustomed to media control, these governments fear losing it to the open internet.” — Vint Cerf

Changing who pays for the internet, and when

The second major reason to be concerned about the ITU’s plans is less philosophical and more technical: since one of the core functions of the group is to set policy around how telecommunication networks connect to each other — and the terms on which those connections occur — many of the proposals before the committee involve extending the same kind of approach to the internet. If you’ve ever experienced the nightmare of international roaming charges for a cellphone (which one estimate says can be more expensive than getting information from Mars), you probably have a sense of why this could be a bad thing.

As my colleague Stacey Higginbotham explained recently, the internet is in many ways a loose collective of national and trans-national networks — all of which connect to each other and share information based on what are called “peering” arrangements. These relationships are based on mutual advantage, and therefore they don’t usually involve money changing hands. In a very real sense, the internet is the first peer-to-peer network, and that has had a tangible impact on everything from telecom competition to the startup ecosystem, since it keeps the costs of cloud computing low.

The internet works, it doesn't need to be regulated by ITR treaty. If it ain't broke, don't fix it. #WCIT12 #EU on.ft.com/WvBFgV
—
Neelie Kroes (@NeelieKroesEU) November 29, 2012

The threat that the ITU poses is that a number of member states want to move to a system called “sender pays,” which is more like how telecom networks operate. The main reason they want to do this is that it would mean a significant new source of revenue for some countries — countries that believe they are currently helping to subsidize the growth of the internet for others, without getting much from it themselves. As a recent report from the OECD argues, making this change could derail many of the benefits that the global economy gets from the internet.

Among other things, this could jeopardize the principle of “net neutrality,” which is designed to ensure that data flows without consideration for who produced it or how much they paid a specific network to carry their bits. As MIT’s Technology Review has pointed out, the Internet Society — a non-profit co-founded by Cerf — has called the proposal an attempt to continue the “scams and arbitrage” that plague the traditional communications model, and says that the model the ITU is considering runs the “serious risk of fragmenting the internet.”

Whether the UN body accepts any of the recommendations or proposals that have been submitted to it remains to be seen, but unfortunately — given the secrecy with which the negotiations are being conducted — we may not know the answer until it is too late.

Post and thumbnail images courtesy of Flickr users Jason Parks and Ryan Franklin

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

How Cisco wants to make big data a community affair

Derrick Harris — Fri, 07 Sep 2012 20:24:57 +0000

Updated. Cisco has a plan for turning big data from a technological movement to a business model that everyone can get behind, and it’s hinged on the notion that large companies can serve as centralized big data repositories that will spur innovation across entire ecosystems. The company lays out its vision in a new whitepaper called “Unlocking Value in the Fragmented World of Big Data Analytics: How Information Infomediaries Will Create a New Data Ecosystem,” which it will release to the public soon.

The paper’s authors from Cisco’s Internet Business Solutions Group argue that big data is a movement rife with possibilities but that’s still out of reach for many companies. During a recent interview, co-author William Gerhardt explained the problem as one of resources and standards. Small companies might not have the innovative minds or high-end technologies in-house to do big data right, he said, and every company large and small suffers when valuable data is locked inside individual companies’ databases or is made available in niche formats.

The solution to that problem, they claim, is the formation of so-called “information infomediaries” that will take care of the heavy lifting of aggregating, standardizing, packaging, securing and processing data from a variety of sources. Gerhardt compared big data ecosystems to other complex ecosystems such as credit card processing and banking, where infomediaries take the reins and provide centralized services to nearly everyone in their industries. We need to simplify this ecosystem or it will fail, he told me.

At the least, it will stall in the early stages of Cisco’s big data evolution chart. In that model, Wave 1 is simple analytics contained within departments and Wave 2 is advanced, actionable and company-wide analytics. Wave 3 is the holy grail — an industry-wide ecosystem where companies and consumers can share data freely and expect a more-predictive analytic experience. According to Gerhardt, most businesses today are stuck in Wave 1.

Chris Osika, another member of Cisco’s IBSG division, predicts these big data ecosystems will actually help mitigate some of the privacy pitfalls that companies face. With the involvement of legal scholars and some creative cooperation among companies, they could do a lot to offer rich services to customers without resorting to “radioactive” practices such as simply selling customers’ data. Of course, as we’ll discuss in some detail at our Structure: Europe event in Amsterdam next month, wide-ranging privacy and data sovereignty laws mean creating international ecosystems could be a tough chore.

And if you’re reading this thinking it odd that Cisco doesn’t appear to have a horse in this race, think again. The whitepaper highlights the importance of the network in terms of providing context across the local networks, the internet and the Internet of Things, and then being able to take action through the various control points along the way. The team’s next step is really digging down to figure out the best way to use network intelligence to spur companies from the first wave to the second wave, Gerhardt said.

In fact, the network is in some ways what provides the most value to Cisco’s proposition given the fast-growing collection of data marketplaces and cloud-based big data services that already make it easier to access resources and analyze data sets. For latency-agnostic applications, “the cloud becomes the great equalizer,” Gerhardt said. But Cisco envisions a world of low-latency applications analyzing and acting on data at network speed, which might require faster networks and techniques such as edge calculations and keeping business logic inside routers.

That would be something, alright, but we’re a long way off.

Update: Cisco’s whitepaper is now available here.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Huawei knocks off Ericsson as world’s biggest telecom vendor

Kevin Fitchard — Tue, 24 Jul 2012 17:54:52 +0000

Huawei on Tuesday reported revenues of 102.7 million renminbi, or U.S. $16.1 billion, for the first six months of 2012. That would seem like a perfectly ordinary quarter for the giant and growing Chinese telecom vendor, but there is something particularly significant of about this earnings report. According to some quick calculations made by Light Reading’s Ray Le Maistre, Huawei’s sales have surpassed Ericsson’s, making the privately held company the largest telco infrastructure maker in the world.

Sweden’s Ericsson brought in $15.25 billion in the first half of the year, putting it $850 million shy of its Chinese rival. That may seem like a lot, but currency exchange rates differences between the Chinese yuan and the Swedish kroner have a big impact. The two also have different portfolios. Ericsson is still by far the largest cellular infrastructure maker in the world, while Huawei has sizable handset and enterprise businesses. Ericsson no longer has the revenues from its handset joint venture with Sony, but it did get a big sales boost this year from its recent acquisition of network systems vendor Telcordia.

Ericsson would surely argue it sells more actual telecom network gear than Huawei, but one thing is certain: this race isn’t over. Both companies are growing despite the poor global economy, and as they continue to land more contracts and currencies continue to fluctuate, they likely will keep leapfrogging one another. Huawei and Ericsson are both well ahead of their next closest competitors, Alcatel-Lucent and Nokia Siemens Networks(si).

The amazing thing is that Huawei has risen to global network prominence despite having almost no impact in the U.S., which along with China are the two most important infrastructure markets in the world. Huawei has some handset deals to sell carrier-rebranded smartphones – the biggest of which is for T-Mobile’s next generation of MyTouch phones – but it doesn’t have a single major network equipment contract to its name in the U.S. Meanwhile Ericsson has its fingers in every major network build of the Big 4 carriers – and most of the smaller contracts as well.

Huawei attributes this to ingrained prejudice in U.S. government circles against a Chinese vendor building the country’s sensitive communications networks. As a privately held company, Huawei lacks the transparency of its competitors, which all trade publicly on major global exchanges. Alleged links between Huawei and China’s People’s Liberation Army have led the U.S. government to block government contracts and acquisitions of domestic companies. Huawei has denied such links and has even invited a U.S. investigation to assuage any security concerns.

In a recent interview, Huawei external affairs VP in the U.S. Bill Plummer said those sinister perceptions of Huawei have cost it a huge amount of business in the U.S., even though European and Canadian carriers haven’t shied away from dealing with the vendor. Plummer said Huawei was on the verge of becoming the third supplier in Sprint’s LTE contract and CDMA network overhaul, but politics got in the way (the contract went to Samsung).

“We were the most competitive offering for Sprint in terms of technology and total cost of ownership, but non-market forces dictated the result,” he said.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Here’s what our web addiction looks like in 2016

Stacey Higginbotham — Wed, 30 May 2012 12:00:27 +0000

We’re on pace to generate 1.3 zettabytes of data in 2016, about four times more than we create today, according to the latest data out from Cisco. To put that in perspective, Cisco helpfully tells us that’s more than 38 million DVDs streamed in an hour. Or, you can think of it as a 1 followed by 21 zeros.

The telecom gear maker offered up its fifth annual assessment of future broadband growth on fixed, managed and wireless networks around the world Wednesday. And to no one’s surprise, as individuals, households and countries we’re just going to keep boosting our broadband use.

Around the world last year people generated 30.7 exabytes of data per month from a total of 10.3 billion connections. That’s a lot until you compare it to Cisco’s projections of the world generating roughly 110 exabytes per month from 18.9 billion connections. That’s a ton of growth, so what do the stats really tell us?

More web use, more users and more connected devices.

Broadly it tells us the growth comes from existing users doing more online while toting more devices, and also from people around the world without connections today who are joining the World Wide Web. Cisco estimates that on average, individuals generated an average of 11.5 gigabytes of data per month. That’s a lot until you compare it to Cisco’s projections of an individual consuming an average of 32.3 GB per month. And as the chart below shows, we’ll be online and so will our vehicles and homes.

Oh, the devices you will have!

Part of what will keep us hooked on the web is a better experience thanks to faster speeds. Global web speeds will rise from 9 Mbps to 34 Mbps. Unfortunately, 100 Mbps speeds and fiber to the home will still be a global rarity five years from now. Cisco estimates only 3 percent of web users will have 100 Mbps connections.

We love the web and Wi-Fi, but what about data caps?

With Cisco predicting average household web use reaching 150 GB per month globally in the unspecified future (beyond 2016 is as specific as the spokesperson could get), and given that U.S. residents are predicted to generate the most IP traffic of any country at 22 exabytes a month, Comcast’s 20 percent boost to its data cap this month had better be the first of many inflation-related upgrades. And they had better come a lot faster. It took Comcast 5 years to go from a 250 GB per month cap to the 300 GB per month cap. In five years Cisco expects North American traffic to increase by three-fold (this includes Mexico and Canada).

And even though mobile growth is skyrocketing and will grow 18-fold in the next five years, it’s rising from a smaller base. Meanwhile, Wi-Fi is going to pick up a lot of the mobile network’s slack because of costs, caps and the growing availability of Wi-Fi networks. Cisco estimates that by 2016, over half of the world’s Internet traffic is expected to come from Wi-Fi connections.

We’re going to need a lot more servers, storage and other equipment to both provide content and store all this data we’re sending, creating and consuming. The network is the transport layer for all those bytes, but machines still create and store them. And really, that gear had better be more energy efficient than the stuff we’re using today, or we’re going to run out of plants to power them.

I just can’t get enough.

For those who care to dig in deeper, here’s Cisco’s web site discussing the survey, as well as our stories covering the first VNI report back in 2008 and another one in 2010. Cisco isn’t doing all that badly when estimating traffic data. In the initial 2007 release of the forecast, Cisco says it projected an overall IP traffic volume of 28.4 exabytes per month for 2011. The actual volume in 2011 was 30.7 exabytes per month, or about 7 percent higher than was projected five years ago.

Our broadband networks are driving our computing, our economy and even our scientific discoveries. Cisco is showing us exactly how big that network is likely to become. It’s pretty awesome.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

CES 2012: a recap and analysis

Thu, 02 Feb 2012 23:40:31 +0000

This year’s CES was the biggest in the show’s 44-year history, boasting 15 miles of exhibit hall aisles, 3,100 booths and 153,000 attendees. The Kerton Group sent three delegates to CES to scout out new products, listen to keynotes, watch announcements and get tips from insiders. This report, which bundles those findings together, serves as an outline of the major launches and overarching trends at CES (think smartphones for $0, Androidification and connectivity) as well as an analysis of what those developments mean for the larger consumer electronics picture. Companies mentioned in this report include Apple, Tesla and T-Mobile. For a full list of companies, and to read the full report, sign up for a free trial.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

The cloud shouldn’t be an over-the-top service

Stacey Higginbotham — Tue, 06 Dec 2011 23:00:37 +0000

Joe Weinman at GigaOM Structure 2011

The public Internet and the cloud shouldn’t mix, according to a paper out today from Joe Weinman of HP. Cisco seems to agree, if Tuesday’s announcement of its CloudVerse suite of products is any indication. A growing number of endpoints, the multiple services built within web applications, and the infinite variety of demands made on any web-based service mean the network can’t be trusted to run over the top.

The network is the cloud, so it needs to be agile, smart and billed based on usage.

Instead, the industry will need to move to pay-per-use, dynamic networks where possible to improve the economic benefits of cloud scenarios and deliver defined quality-of-service for applications that will require low latency, argues Weinman. Weinman, who moved over to HP from AT&T last year, is a deep thinker on the economics of cloud computing. He also argues that bandwidth will eventually be charged on a pay-per-use model for both consumers and enterprises.

He makes a good case for the importance of a smarter network in the context of delivering cloud services, something Cisco’s CloudVerse announcement Tuesday also supports. CloudVerse basically organizes Cisco’s existing networking products for the data center and links them back to the networking gear already in carrier and service provider networks, with the idea being that an intelligent network can take the fuzziness out of managing applications in the cloud.

Complex apps and infinite endpoints make quality of service more important.

It’s true that applications are growing more complex and relying on more protocols to deliver a variety of services over the web. Take, for example, an application like Google+. There are real-time streaming elements, a video conferencing set-up and document sharing. Each different element requires different levels of network quality, which is why Weinman argues for networks that run faster, not just on a megabit-per-second basis, but also with less latency. From the paper:

Human performance studies show that 200 to 250 milliseconds is acceptable for multimedia conferencing and collaboration applications. However, interactive tasks such as keystrokes and mousedowns must be responded to within about 150 milliseconds [10], and emerging online games require even lower latencies.
There are also empirical results showing the importance of low latency not just in terms of user experience, but in terms of revenue. Lower latency directly correlates with increased revenue [11].

Add in the complexity at the end point in terms of the number of devices that connect to the network and it gets worse. Sensor networks, plus more devices per person and more concurrent streams coming in per device (as in personal video recorders) require more bandwidth. It also requires more intelligent bandwidth that can allocate resources and deal with emergent effects like in-office or in-home congestion and odd spikes in traffic in case of unexpected events. For example, a pipe breaking in a sensor-equipped home in the middle of the afternoon when the house is empty may create a sudden spike in traffic as humidity sensors activate, power gets shut in certain areas and you check in via a home camera system to see why your home network is going crazy. But because that’s an unexpected spike in a normally dull time, will your service provider have the bandwidth capacity to meet that event?

Of course, there’s something on cloudbursting and software-defined networks.

Weinman also offers the Holy Grail of true cloudbursting as an example where adding network intelligence makes it easier to scale a workload from one data center to another in times of peak demand. He lists five ways of doing this, beginning with the simplest idea of dividing up tasks between various clouds, which requires little to no network intelligence. He concludes with a network that can push a huge amount of data as needed and very quickly, but which would require infinite bandwidth. Since this last approach is impractical, he suggests providing pay-per-use bandwidth as the easiest way to instantly replicate data while keeping costs in line.

To help deliver the type of fine-grained control that intelligent networks will need, Weinman believes software-defined networks, such as those created using protocols like OpenFlow, are a way to add intelligence and flexibility. Using open protocols to create the networks are a good way to make sure that the added intelligence doesn’t act as a way to lock in users. Weinman also covers additional topics that will require research in bringing forth these new networks for cloud computing, and I highly recommend folks check out his paper.

This sounds great; so how do we co-opt it to sell products?

So what does this have to do with Cisco’s marketing effort around CloudVerse? Essentially, with the suite of products that wrap data center networking in with the networks of service providers for wireline and mobile broadband, Cisco is recognizing that a holistic, intelligent network could be a huge selling point for those concerned about piecing together their own fragmented network elements to deliver web services and cloud services. A quote from the Cisco release sums up the news nicely:

“Until now cloud technology resided in silos, making it harder to build and manage clouds, and to interconnect multiple clouds, posing critical challenges for many organizations,” said Padmasree Warrior, Cisco senior vice president of engineering and chief technology officer. “Cisco uniquely enables the world of many clouds – connecting people, communities and organizations with a business-class cloud user experience for the next-generation Internet.”

Cisco and Weinman are not alone. Alcatel-Lucent recently outlined its vision of as service provider cloud that adds intelligence to the network in a way that many enterprise and business customers will find appealing.

Could someone build a fully functioning network without resorting to all-Cisco gear, or perhaps even Weinman’s view of the intelligent network? Yes, but it takes skill and dedication that places such as Google, Yahoo, Facebook and other webscale operators have, and other companies just don’t seem to want to bother with.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

For science, big data is the microscope of the 21st century

Stacey Higginbotham — Tue, 08 Nov 2011 23:03:14 +0000

Johns Hopkins is taking a $1.2 million grant from the National Science Foundation to build a 100 gigabit per second network to shuttle data from the campus to other large computing centers at national labs and even Google. The network will be capable of transferring an amount of data equivalent to 80 million file cabinets filled with text each day.

The head of the project, Dr. Alex Szalay, detailed the plans, which include gear from networking gear from Cisco, Arista and Solarflare; Nvidia GPUs; and 66,000 x86 cores. That’s on top of the actual fiber that will connect a new, 1-megawatt data center inside the physics building to regional Mid-Atlantic Crossroads research and engineering network at the University of Maryland.

The new data center at Johns Hopkins, awaiting its 100 Gbps backbone.

That connection will be the 100 Gbps element funded by the NSF, and the Mid-Atlantic Crossroads network connects out to Pittsburgh and then onto Chicago via other 100 Gbps networks that are growing in number across the country. Inside the campus, Szalay, who is the alumni centennial chair in physics and astronomy at Johns Hopkins, is setting up a 40 Gbps network between buildings that deal with lots of data such as the medical and computer science hubs. “To keep looking at big data sets we have to move the big data to a location where we can analyze it, and the stumbling block is [data sets of more than] 100 terabytes because of the speed of the network,” Szalay said.

He ascribes this massive amount of data to the emergence of cheap compute, better imaging and more information, and calls it a new way of doing science. “In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st-century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay said.

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing which will lead to new advances in astronomy and physics, much like the microscope’s creation in the 17th century led to advances in biology and chemistry. When thought of in that light, the creation of 100 gigabit per second research network at Johns Hopkins becomes not just a fast network, but an essential tool for research and discovery, an essential component of the 21st-century microscope.

For example, he described trying to send a 150-terabyte chunk of astronomy data for analysis to Oak Ridge National Lab in Tennessee as “painful” because of the limits the 10 gigabit connection present between the university and the national lab. When he looks ahead 10 years and anticipates a colleague’s next-generation astronomy project currently underway that Google is supporting with 14 million compute hours, he believes it could generate 100 petabytes of data.

If that kind of data avalanche is a mere decade away, it appears our faster networks can’t come soon enough. It’s a good thing Johns Hopkins expects the 100-gigabit network out and the 40-gigabit intra-campus network will be functioning in April.

Image courtesy of Flickr user RinzeWind.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

Why the world of scientific research needs to be disrupted

Mathew Ingram — Mon, 31 Oct 2011 21:32:34 +0000

Traditional media players such as newspapers, magazines and book publishers often get criticized for being slow to change and uninterested in technological progress, but as we’ve pointed out before, there is another world that makes these industries look like the most enthusiastic of early adopters: namely, academic research. Award-winning quantum physicist Michael Nielsen says that the closed and disconnected nature of most research is holding back scientific progress in important ways, and that we need to foster a new kind of “networked science” if we want to make new discoveries faster.

Nielsen makes this argument in an op-ed piece written for the Wall Street Journal, which in turn was adapted from a book he published earlier this month called “Reinventing Discovery: The New Era of Networked Science.” The author is recognized as an authority on quantum computing — having written one of the premier texts on the topic, as well as about 50 scientific papers for various journals — and was a faculty member at the Perimeter Institute for Theoretical Physics in Waterloo, Ontario. But chose to put his quantum computing work on hold in order to write the book, because he felt so strongly about the need for more collaboration and what he calls “open science.”

Some scientists collaborate openly, but many do not

The physicist describes a number of successful collaborative efforts that have made real progress in scientific research, including one called The Polymath Project, which started with a simple blog post by a mathematician at Cambridge University who wanted to see if he could get help with a problem. Within a matter of hours, comments had poured in from mathematicians, a high-school math teacher and others around the world, and within six weeks the problem had been solved. Unfortunately, as Nielsen points out, this kind of collaborative effort is rare — and not just in mathematics. As he explains:

If you’re a scientist applying for a job or a grant, the biggest factor determining your success will be your record of scientific publications [so] you devote your working hours to tasks that will lead to papers in scientific journals. Even if you personally think it would be far better for science as a whole if you carefully curated and shared your data online, that is time away from your “real” work of writing papers.

As Nielsen and others have pointed out, this reality stifles a lot of scientific research, not to mention slowing down what research does occur — since it has to take place in a tiny number of peer-reviewed journals (the ones that your academic superiors see as worthy), which take months or even years to publish. And as George Monbiot pointed out in a rant against academic publishing in The Guardian earlier this year, those journals are also only available to other academics, often at unreasonably high prices — even if the research that the article is based on was funded by public money, and much of the peer-review and editing that went into it was done free of charge.

Part of what is disrupting scientific research is the simple fact that the web exists, and the “democracy of distribution” (as Om likes to call it) that digital-media tools have created — the same ones that allowed the high-school math teacher to help solve the Polymath Project problem, even though he isn’t a member of any of the prestigious societies or journals that usually deal with such things. It’s more than a little ironic that many scientists still don’t use the internet much for collaboration, when the network was originally created in part to help universities share research more easily — although projects like Mendeley are doing their part to try and change that.

The network is changing the way knowledge works

David Weinberger, a fellow at Harvard’s Berkman Center for the Internet and Society and co-author of a number of books including “The Cluetrain Manifesto,” has his own take on networked knowledge in a new book called “Too Big to Know,” which is to be published later this year. Weinberger argues that the way we structure and achieve knowledge itself is being changed by digital networks, and that much of the existing ways in which knowledge is written down and maintained — from journals and peer review to libraries and copyright — is driven by the needs of a world based on paper:

If your medium doesn’t easily allow you to correct mistakes, knowledge will tend to be carefully vetted. If it’s expensive to publish, then you will create mechanisms that winnow out contenders. If you’re publishing on paper, you will create centralized locations where you amass books… Traditional knowledge has been an accident of paper.

So how do we disrupt the academic-research business the same way that Amazon and the web have disrupted book publishing, or blogs and The Huffington Post have disrupted newspapers? Nielsen doesn’t have any silver bullets, but he does suggest that government agencies funding research should require that those submitting papers must provide their research free of charge (the National Institute of Health has started doing this with research it funds or supports).

Nielsen also argues that scientists themselves need to start bucking the system and supporting open research, as some — including Microsoft researcher Danah Boyd — have tried to do. Unless scientists and researchers start to put the interests of collaboration and “open science” ahead of their desire to be promoted or win tenure, he says, the system will not change, and experiments like Project Polymath and others he describes in his book (such as Galaxy Zoo, which allows non-scientists to help identify interstellar phenomena) will continue to be the exception instead of the rule.

Post and thumbnail photos courtesy of Flickr users Jeremy Mates and Sandy Honig

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.