Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• 12 tech leaders’ resolutions for 2012
• NewNet Q4: Platform mania and social commerce shakeout

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• Infrastructure Q2: Big data and PaaS gain more momentum

Those optimistic about 2013 point to the second half of the year (now only four months away) as when we’ll begin to see some stronger growth. In their view, Europe will be better, the U.S. will be stronger, China will begin to reaccelerate in earnest, currency will be stable, and (perhaps most importantly) the year-over-year (YoY) comparisons between 2012 and 2013 will be easier because mid-2012 was so weak. Moreover, as EMC CEO Joe Tucci stated during the company’s recent earnings call, “You can’t starve IT for too long,” which suggests that better times must get rolling again. On the other hand, CEO John Chambers’ statement in the Cisco earnings call, that perhaps 2012 represents the new normal, might be a better indicator of what we can expect in 2013. We’ll see.

That said, here’s what we’re expecting for 2013.

Table 1. 2012 and 2013 worldwide IT spending (in $ billions)

Overall, smartphones and tablets will lead the way, with a 5.6 percent worldwide IT growth rate in 2013. While Apple’s tepid forward guidance in its recent earnings call and its implications for smartphones and tablets gives us pause, we expect the category to continue to lead, with growth in the 20.4 percent range in 2013. In hardware, 2013 will look like 2012, with smartphones and tablets buoying the otherwise-weak spending in the PC, peripheral, and server segments. (This will be offset somewhat by stronger storage and network hardware spending.) Software continues to flourish — especially solutions-related software like SaaS-enabled customer-relationship management (CRM), supply-chain management (SCM), and industry vertical systems like health care. This sector will continue to grow in 2013. Finally, services has experienced a punishing 2012, and this area will only look modestly better in 2013.

A methodological note: We’ve reevaluated our model and determined that we have underweighted business-process outsourcing (BPO) as a service category. As such, we are restating 2012 and 2013 growth rates to accommodate a higher weighting of BPO in our services spending. The net is that growth in services and all spending are a bit more robust, since BPO is growing faster than the aggregate of traditional IT services. The full methodology can be found at the end of this report.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• IT spending update, third quarter 2012
• IT spending update, fourth quarter 2012
• Work media tools in 2012 and beyond

To build the internet of things we’re going to need a lot of chips — orders of magnitude more than we have in use today. Generally those chips will fall into three categories, and each of those categories is poised to become a booming business with a lot of volume and room to grow. Let’s break it down:

Connectivity: This one is a no-brainer. If we want things to connect to the internet, we’re going to have to put radios in them. It may be Wi-Fi, Bluetooth, Z-wave, ZigBee or even a 3G or 4G cellular standard (or all of the above) but there has been and will continue to be a land grab for radios among the big chip companies. The rise of connected devices is the reason Qualcomm bought Atheros back in 2011 and the reason little known microcontroller company Atmel purchased Ozmo Devices in December.

We’ll also see new products aimed at integrating radios together, not just from Broadcom — the king of radio integration — but also smaller companies such as Redpine Signals, Altair and others. And these radios will be going into more devices. Just a quick scan of Kickstarter or Indiegogo shows a plethora of home gateways, Wi-Fi enabled devices and sensors that have radios integrated from a variety of vendors. A report from the OECD on the internet of things estimates that a family of four will go from having an average of 10 devices connected to the internet now to 25 in 2017 and 50 by 2022. Every single on of those will have a radio — or multiple radios.

Control: These chips are the brains of the operation. But unlike in the personal computer or server market, where Intel and AMD fought for dominance (more truthfully, AMD tried to at least achieve profitability), or the smartphone market where Qualcomm has taken out competitors ranging from Texas Instruments and Freescale on the application processor side (leaving Apple, Samsung, Broadcom and Mediatek standing), this market has a much wider variety of players known for their embedded processors and microcontroller. The one name that spans all of these industries is ARM.

At the low end, microcontrollers can range from 8-bit processors that manage setting on your microwave to higher-end chips inside a set-top box. Companies like Freescale, Texas Instruments, Atmel, Intel and STMicroelectronics all are pushing their microcontrollers (MCUs) inside the internet of things. The variety of use cases and devices inside connected devices mean some gadgets will need more power savings than performance or merely just a cheap 32-bit chips designed for a more industrial application. Many of these companies have an advantage for the internet of things because they are used to supporting a wide variety of end products with their firmware and sales teams.

They have designed their chips to be modular. If the bigger players want to play here they will have to build out multiple lines of chips with differing performance specs that can be supported across a wide range of end devices. That’s very different from building out a line of chips with slightly different specs all designed for servers. I bet a few of the big vendors, especially on the connectivity side, might try to acquire this knowledge.

Botanicalls moisture sensing system.

And since many of these sensors will be integrated onto small packages with radios and maybe even MCUs there will be a lot of value for a company that can pop all of the above onto a system on a chip — it’s cheaper, smaller and more power efficient. So consolidation will happen within these categories as well as across them as more devices get online and we ask them to share more information about their environment.

So be they MEMs, microcontrollers or radios, there’s a lot of silicon (or maybe gallium arsenide) inside the internet of things. And the types of chips required will stretch the silicon industry — that has been primarily focused on keeping up with the performance requirements of Moore’s Law — into new directions. Power savings, integration and size will matter when it comes to connected devices more so than the all out race for performance that has dominated the chip industry for decades.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing’s impact on chip and hardware design
• Cloud computing infrastructure: 2012 and beyond
• Who and what to watch in the new era of the living room

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• How the mega data center is changing the hardware and data center markets
• Migrating media applications to the private cloud: best practices for businesses
• 12 tech leaders’ resolutions for 2012

The network, he contended, should be “stupid,” carrying bits from point A to point B, and not doing much else. Functionality was best delivered by intelligent endpoints interacting over a dumb network. As he foresaw, the interoperability benefits of a ubiquitous protocol like IP, which has now worked itself into our smartphones, tablets, and TVs – not to mention everything from electric meters to light bulbs – cannot be denied. And, thanks to Moore’s Law, even preschoolers can have hundreds of GigaFLOPS at their disposal for less than the price of a swing set.

Of course, 15 years is a long time, especially in the field of computing and communications. So the question is, does Isenberg’s line of thought still hold true? I would argue that, rather than stupid networks, we’re entering an era of “pervasive intelligence,” where endpoints are intelligent, but the network can be as well. Networks can be smart. Tunable. Programmable.

A simple analogy: Suppose you’d like to enjoy a tropical beach vacation, but are constrained by a fixed budget. Anywhere supporting your Vitamin D requirements would do. You might start by comparing resort prices in, say, Bali, Phuket, St. Tropez, and South Beach. But naturally you wouldn’t just factor in the price of the stay; you’d also factor in the cost of transport. The best decision then wouldn’t necessarily be the lowest cost resort or the lowest cost plane fare, but whatever led to the lowest total cost. To put it another way, you wouldn’t just optimize the endpoint or the transport, but would consider both together. Consider that when we check out at the grocery store, we don’t just select the most energetic cashier, but also consider the length of the queue.

Vacation planning and grocery shopping help illustrate an experimental algorithm designed at Stanford, described by Software-Defined Networking / OpenFlow icon and Stanford Professor Nick McKeown, in a YouTube video. The experiment, run on the large scale GENI (Global Environment for Network Innovations) testbed, contrasts two approaches to load balancing, or the distribution of work across multiple servers to minimize response time and maximize throughput.

Source: YouTube

As can be seen from the chart above (a still taken from the YouTube video), random load balancing (the red line) has dramatically higher worst-case response times and variability than selecting a path simply based on lightest real-time network congestion (the green line).  Even better results would be generated by an algorithm which also accounted for server load.  As McKeown explains, “ideally, [a] request would be sent over a path which is lightly loaded to a server which is lightly loaded. In other words, we would jointly optimize the combination of the path and the server… .”

The reason McKeown reviews this example is to illustrate the power of software-defined networks and existing testbeds to accelerate innovation. As he puts it, “The point here is…a graduate student was able to take an idea, and within a few weeks, put that into a national network, run real traffic over it, … demonstrate it to others, and then hand it to them and say here’s the code.” In addition I think this particular experiment also points the way to a world of intelligent endpoints collaborating with an intelligent network to achieve something neither can do as well alone. As McKeown deduces, joint optimization would generate the best results.

A variety of technologies that enable network smarts to contribute to overall end-to-end performance and ease-of-use are emerging. Consider HetNets – heterogeneous networks that span Wi-Fi and 4G, for example. Enabling seamless handoffs between the two benefits from network intelligence. Or consider peer-to-peer file sharing. Rather than fetching a copy of a file from a location halfway around the planet, emerging approaches such as ALTO (application-layer traffic optimization) will be able to help select a more efficient location hosting that content nearby. (Being a locavore – consuming locally – can be good not only for produce, but for information products.) Moreover, such optimization can be good for users, network service providers, and over-the-top service providers.

Ultimately then “intelligent endpoint, stupid network” vs. “stupid endpoint, intelligent network” is a false dichotomy. As the Stanford work tantalizingly suggests, the best of all possible worlds may actually be smart endpoints harmoniously coexisting with a smart network.  Or perhaps even other configurations; consider the case of light bulbs and netbooks, where “stupid” endpoints access “smart” endpoints – either through today’s IP networks or tomorrow’s software defined networks, built of “dumb” switches directed by intelligent control planes. (Or, a variety of other options with unevenly distributed intelligence that come together to best deliver some particular functionality.)

New algorithms under investigation by researchers are even moving beyond networks and endpoints into additional concerns, such as power. For instance, some approaches dynamically migrate and consolidate virtual machines within a data center to enable freed up physical hosts to be powered down; others move workloads across data centers where the instantaneous cost of power is lowest.  Some might even argue that cloud computing itself demonstrates that the “network is the computer,” where services are delivered by a distributed intelligent fabric.

Moore’s Law effects mean that the cost of intelligence is dropping. And so we may as well increasingly leverage it in today’s digital economy wherever there is a net return: in the endpoint, in the network, or both.  This suggests the fall of the stupid network, and the rise of pervasive intelligence.

Joe Weinman is a senior vice president at Telx, the author of Cloudonomics: The Business Value of Cloud Computing, and a regular guest contributor to GigaOM. You can find him on Twitter @joeweinman.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Survey: How apps can solve photo management
• Cloud and data first-quarter 2013: analysis and outlook
• How fourth-quarter 2012 will affect IT spending in 2013

In 2005, Hoffman’s mother was diagnosed with breast cancer. His grandmother had died of the same disease . His mom thought it was a death sentence. Hoffman, then working as a doctor, reoriented his life to help her fight the disease. But instead of drawing on his medical training, he tapped his interest in computing — specifically the small hosting company he had started as a side project to help some of his friends build out new web services such as WordPress (see disclosure) or 37 Signals.

He quit his day job as a cancer pathologist, moved from San Diego to San Francisco, and spent the next 18 months working with his mom’s treatment team to develop a custom-drug program.

Her treatment was “unorthodox” because it combined research on specific cancers with genetic analysis on his mom. That genetic analysis required a lot of computing power optimized for visualizing data — power that Hoffman supplied with his hosting company, Joyent.

For the first time since he was about 16, Hoffman was focused on one job — building Joyent — instead of two or three. That, and saving his marriage and his mom’s life. Despite long odds, his mom survived. His marriage didn’t.

Jason Hoffman racking servers.

As for Joyent, once Hoffman turned his full attention to the company, he decided it had to be bigger than hosting a few blogs or helping a dozen entrepreneurs. “If this is what I’m going to do … It needs to be more of a change-the-world type of thing. The plan became how could someone build a modern systems company and deliver it as a service?” Systems, as in giant clusters of servers that deliver all those games and crunch genomics data all at the same time.

Our raging addiction to the Internet and the demand for computing everywhere we go is forcing a kind of reckoning in the IT space. Much like widespread automobile adoption forced a rethink of the transportation system, our dependence on the internet is forcing us to rethink the way we build computers.

The next generation of cloud computing and data processing infrastructure will need to have flexible pools of storage available near powerful processors with a superfast and flexible network bridging the two. It’s massively distributed and looks more like a supercomputer than a rack of servers topped by a switch. Google knows it. Intel knows it. VMware knows it. And before these companies knew it, Jason Hoffman knew it.

From Army brat to lab rat

Hoffman was born in Long Beach Calf., where his father brought the family after a tour in Vietnam. His father, a soldier, worked as an honor guard burying the dead coming back from the war, and his mother was a nurse. His family moved to Germany when he was six months old. And for the next decade, Hoffman started each year in a new school in a different place in the world, as an army brat.

Despite his itinerant lifestyle, Hoffman did well in school, something he attributes to his voracious reading. During sixth grade, he read about two books a day and wrote them up as book reports, earning the ire of his classmates. Hoffman can still rankle others in the industry — in part because he is incredibly smart and not shy about sharing his opinions or rationalizations.

Earlier this year, Joyent decided to shut down a service called TextDrive, despite having promised early TextDrive users that they would have lifetime accounts. Hoffman, a co-founder of TextDrive, which was then purchased by Joyent, took the vitriol of the angry users calmly, broadcasting snippets of the hate mail he received in a series of tweets. Hoffman later worked with TextDrive co-founder Dean Allen to address those concerns.

“Jason is not a victim,” says Dr. Elie S. Semaan, his former college roommate and now a surgeon at New York Methodist Hospital. “Even when he hits these low points you won’t see him thinking about how bad it is. He just keeps moving forward.”

Hoffman hugging his servers.

It’s just with corporate strategy that he can be dispassionate. At one point during our interview, for example, he described his mom’s cancer diagnoses as “problematic.” Rather than talk about what she means to him or how he felt at the prospect of losing her, he went straight to how he solved the problem.

Reading between the lines as Hoffman talks, he seems to feel that emotions can be an impediment because they can get in the way of clear-eyed solutions. He mentions, for example, the time he spent working in the ER during high school and college.

“It was a nice little crucible to go through. The hospital was in an area where a lot of gangs were, and it was war zone-ish.” Hoffman said. “It wasn’t unusual to see a shooting and two stabbings a night. When you see that, and you’re 18 through 22, you begin to see what is actually an emergency and what is really something tragic. You also see how a team works and learn how to work under pressure.”

Despite not wearing his emotions on his sleeve, Hoffman is a good friend to many in the industry. At home, he spends as many weekends as possible with his three daughters (one with his first wife and two with his second). On Hoffman’s birthday earlier this year, his wife and youngest daughter came to see him backstage at an industry conference bearing donuts, and despite the danger of a sticky suit before he was set to go on stage, he held his donut-eating daughter in his lap.

Yet, even here he is supremely rational. Knowing that work and family are the two most important things in his life, he’s set up a system that supports both. He says he lives equidistant between his office and the airport so he can get home as quickly as possible from work or a trip.

Computing and cancer aren’t too far apart

Hoffman’s beginnings in medicine primed him to rethink computing. From early in his career as a cancer researcher, Hofmann was playing with computers. First it was giant SGI machines to help handle large data sets and images, but later it became Linux boxes and even mini supercomputers he built himself. Some of his initial research involved looking at people with cervical, breast and certain types of skin cancer and understanding genetic markers that predisposed them to these diseases. But the imagery and data sets taxed his machines.

“The problem with computers is that they are built for doing math, but doing math is different than looking at images and moving bits and storing things, so I had general frustrations around storage,” Hoffman said.

With Joyent, he saw the opportunity to fill that void– and provide exactly the type of computing we’re seeing with the rise of online video and picture-based services like Pinterest and big data. And he thought about ways to do it on the cheap to support those burgeoning online businesses.

Hoffman’s drawings on how to scale out WordPress’ infrastructure.

Joyent has raised $150 million, and landed dozens of mobile and ecommerce customers including ModCloth, GiltGroupe, LinkedIn, Digital Chocolate, Adobe and Voxer. Commercial success is good, but with a $150 million investment, the question for Hoffman is whether the company will go public or sell.

On the technical side, Joyent’s goals are much clearer. Hoffman says he will soon be able to deliver an exabyte of storage — that’s the equivalent of 250 million high-def movies — with only 10 megawatts of power and $2 million. That’s a big brag, since the traditional thinking is that delivering an exabyte of storage would take around $200 million.

For all its success, though, it’s still unclear whether Joyent will become the next big thing, or a Sun Microsystems– that is, home to brilliant technology that is unable to dominate the market. Joyent’s business of selling a private cloud based on a fundamentally different compute architecture means it uses a different operating system to deliver the best performance over a massively distributed system.

Giant companies generally don’t want to adopt new OSes, and while Joyent offers hardware to service providers in on-premise boxes so customers can choose their provider, its other big problem is the competition. The company is basically trying to carve out a place as the infrastructure equivalent of an Aston Martin to Amazon’s Toyota. And it’s hard to fight against the commodity pricing that Amazon can offer.

Hoffman sees this as his chance to make a mark on the compute world, and he’s pushing forward. And perhaps that’s his style — pursuit of a lofty intellectual goal and little tolerance for any drama that might slow the ship down. It looks like arrogance, but if he winds up developing tomorrow’s distributed compute architecture, no one will care.

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOm. Om Malik, founder of GigaOm, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• How tomorrow’s mobile-centric data centers will look
• 12 tech leaders’ resolutions for 2012

At its core, the research is attempting a big change — building out communications networks that won’t need to convert information back to electronic bits. It’s all photons, no electrons. Eliminating that conversion would cut down on latency, energy and eventually cost if this research can be commercialized. Of course, today’s networking infrastructure is built around zipping electrons over copper and other conductive materials, so that would need to change, but light in the form of fiber optics is encroaching because of its greater capacity and speed.

Currently, long-haul pipes and some last-mile access networks use fiber. On those networks, the bits travel as far as possible in the form of light waves before getting converted back to electronic information to be sent further or read by the communications or data center gear. But as our computing and broadband networks become more distributed and need to share more and more information in real time, networks are getting bogged down.

That’s why startups like Lightwire (purchased by Cisco) Kotura and Luxtera are trying to build silicon photonics to bring light onto the silicon chip for faster on-chip communication. Even companies such as Plexxi are trying to bring fiber gear into the data center to add capacity and speed to scaled out networks. In broadband networks, I’ve covered the move to a terabit age, and companies from Google and Verizon to Ciena and Infinera are all delivering products and services to make that a reality.

That’s the big picture. At the micro level, this research has shown that people can build a device that allows light to interfere enough with existing light waves to allow it to change state. When an electron changes state on a chip, it’s represented as a zero or one — the basis for all of our computer programming. If light can switch physical states, then it too can store data.

From the news release discussing the research:

Glass optical fibers carry many communication channels using different colors of light assigned to different channels. In optical cables, these different-colored light channels do not interfere with each other. This non-interference characteristic ensures the efficiency of a single optical fiber to transmit more information over very long distances. But this advantage also harbors a disadvantage. When considering computation and signal processing, optical devices could not allow the various channels of information to control each other easily…until now.

The net result is faster and more efficient networking, something we can all get behind. Eventually. Currently, the new optical-relay device operates 1 million times per second and the goal is to get it to switch states several billion times per second. However, the current device is fast enough to start using it to connect fiber networks directly to broadband radio networks without the electronic conversion. Here’s a link to the original paper published Thursday in Nature for those who want to learn more.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Survey: How apps can solve photo management
• Opportunities for the future of batteries
• A clouded view of Google Music

While MightyText is clearly an app that one associates with mobile phones — after all texting originated there — it’s really trying to build connective tissue between all of our platforms with broadband as the mechanism to do so. As many teens could tell you, there’s no difference between a text and an IM unless you consider the device. And thanks to IM applications getting loaded onto phones, there’s really no reason to involve your carrier — something that helped precipitate the changes in mobile data plans by AT&T and Verizon.

Mobile first, and everything else feeds into that.

The lesson here is the mobility isn’t a separate platform, it’s the primary platform for most apps and everything else should be built to tie back into it. This is a fundamental shift in worldview, and some companies like Spotify or MightyText get it, and some companies are struggling, such as Facebook. In an interview a few months back, Nagraj Kashyap, vice president of Qualcomm Ventures, explained how he thinks about mobility and how that influences his investments.

“In general on mobile phones, resources are more expensive and attention spans are less,” Kashyap says. “We like companies that are more creative and come at it without a legacy way of thinking. The app is based around the mobile phone: and we have a growing focus on how to leverage the smartphone and not worry about the PC online business.”

First, he’s focused Qualcomm Ventures more on early stage companies and he’s also broadened the geographic focus for Qualcomm Ventures. There are plenty of places in the world where wireline broadband is a luxury, but millions access the web on their smartphones. Given that, places like Brazil, China and India have a mobile first attitude already giving them an advantage. He’s also broadened the concept of what makes a “mobile company.”

Qualcomm Ventures invests in mobile companies, but some of its investments just like some of the QPrize finalists don’t really seem like a traditional mobile play. For example, Cloudessa, one of the finalists, is a cloud security company. But since many of the potential customers of the Cloudessa software are likely enterprise customers worried about their employees working on corporate apps on unsecured personal phones or tablets, Qualcomm’s interest makes sense.

Another area where Qualcomm is stretching the definition of mobile is in its healthcare investments. For years Qualcomm has been preaching the benefits of linking mobility and wireless devices to medical devices and healthcare in general. Although Ubiqi Health, a QPrize finalist, isn’t a novel medical device that takes advantage of connectivity, it does provide a mobile-first dashboard for managing a chronic condition, tying together two of Qualcomm Venture’s investment theses.

Some other elements of Qualcomm’s worldview and what it aims to invest in through its venture arm are companies that take advantage of the mobility of phones to gather data from users and then turn that data into useful services. Examples of this are Waze, which makes crowd-sourced traffic maps or RootMetrics, which measures wireless signal strengths on phones that run its software and then deliver coverage maps. (Update: To be clear, Qualcomm doesn’t have an investment in RootMetrics.) Kashyap is especially interested in using camera phones and augmented reality along with crowd sourcing to build up a repository of imagery that could help push the envelope on computer vision.

In general Qualcomm’s investments and QPrize finalists exemplify Qualcommm’s mobile-first worldview. And once that worldview is factored in, then the devil is in the details. Remembering that bandwidth is expensive and more limited means designing lightweight apps. Shorter attention spans mean that the lifetime of apps may be short-lived if they are entertainment-focused and that productivity apps will have to balance brevity with a business model (this means advertising that requires extra time and space) may be out.

It’s a new world, and Qualcomm is investing at the forefront of it.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Survey: How apps can solve photo management
• How tomorrow’s mobile-centric data centers will look
• Analyzing the wearable computing market

In this case, however, I’m not talking simply about creating and controlling interoperability from the developer level. Tools and services like Dell’s Boomi or IBM’s CastIron have existed for years, and have some success in delivering more flexibility to integration between applications and services. However, these services are focused on solving the developer’s key issues with integration –how to make sure messages move between components based on a process definition and one or more translations, if needed.

The interoperability challenges facing IT operations

But today application operators see a tangental set of problems, and these problems are increasingly becoming difficult to deal with. For the operators, the problem of interoperability has several parts:

• Maintaining interoperability with dependencies.For the developer, the problem of managing dependencies is one of logic—finding the right configuration of code and file dependencies to allow the application to execute successfully. This is largely a static problem, though one that increasingly requires devs to design for resiliency; if one dependency disappears, an alternative method of achieving the task at hand should be attempted instead. For operations, however, the problem is ongoing, as operations has to deal with the reality of why a dependency failed the component or components that depended on it.
• Maintaining interoperability for dependents. The rapid growth of cloud services and APIs, on the other hand, make it operations’ job to deliver availability, performance and consistency of the software systems they operate to those that depend on that software. If you plan on earning business via services delivered via APIs, your operations team has to ensure that those services are there when your customers need them, without fail. Even if you simply provide data via batch files to a partner or customer, that mechanism has to run as the customer expects it to, every time.
• Maintaining interoperability with things operations controls. The other key aspect of operations focus on interoperability has to do with control. There is a variety of responsibility that is inherent in operating systems that interact with one another. The goal of operations, in this case should be to optimize how these systems work together once deployed. Some of that is going back to developers and asking for changes to the applications or data themselves, but often much of that optimization has to do with network and storage configuration, tuning virtualization platforms, ensuring security systems and practices are in place, and so on.
• Maintaining interoperability with things operations doesn’t control. Perhaps the most interesting aspect of application operations in the cloud computing era is the increased need to maintain control of one’s applications in the face of losing control over key elements on which those applications depend. Dealing with upgrades of third party services, handling changes to network availability (or billing, for that matter), or even ensuring that data is shipped to the correct location, on the correct media, by the right delivery service, are all tasks in which operations can only effect one side of the equation, and had to trust and/or respond to changes on the other side.

Complexity makes interoperability difficult

None of this is a shock to most IT operators, but there is one other element that I’ve hinted at before that is creating the rapid expansion of complexity facing operations today, and that is the sheer volume of integrations between software and data elements both within and across organizational boundaries. It’s no longer a good idea to think of individual applications in isolation, or to assume a data element has one customer, or even one set of customers with a common purpose for using that data.

Today we live in a world where almost everything that matters in business is connected by a finite number of degrees of separation from just about everything else in that category. Cloud computing is one driver, but the success of REST APIs is another, as is the explosion of so-called “big data” and analytics across businesses and industries.

We, in business software, exist in large part to automate the economy, in my opinion. The economy is a massive, highly integrated complex adaptive system. Our software is rapidly coming to mimic it.

We need standard operations interoperability

All of this brings me to the opportunity that this interoperability explosion brings to operators and vendors of operations tools alike. If we are going to manage software and data that interoperates as a system at such a massive scale, we need tools that interoperate in support of that system. We need to begin to implement much of what my friend, Chris Hoff, called for five years ago from the security software community:

We all know that what we need is robust protocols, strong mutual authentication, encryption, resilient operating systems and applications that don’t suck.

But because we can’t wait until the Sun explodes to get this, we need a way for these individual security components to securely communicate and interoperate using a common protocol based upon open standards.

We need to push for an approach to an ecosystem that allows devices that have visibility to our data and the network that interconnects them to tap this messaging bus and either enact a disposition, describe how to, and communicate appropriately when we do so.

We have the technology, we have the ability, we have the need.  Now all we need is the vendor gene pool to get off their duff and work together to get it done. The only thing preventing this is GREED.

Amen, Chris. That remains as true today as it was then, as far as I can tell. Only now the scope has exploded to include all of application and infrastructure operations, not just security software. While everyone is looking for standards that allow one tool to talk to another, we are missing the bigger picture. We need standards that allow every component in the operations arsenal to exchange events with any other component, within understood guidelines. That may be as simple as setting the expectations that any operations software will have both an execution and a notification API set.

Another option is a formal event taxonomy and protocol, but that option doesn’t interest me very much. Those standards tend to become outdated quickly and are far too restrictive.

One last thing: John Palfrey and Urs Gasser have written a book on interoperability which I am in the middle of reading. So far, the most interesting aspect of the model they describe is a multi-tiered view of interoperability that supplements data and software interoperability with human and institutional interoperability. The latter two concepts are incredibly important in the new cloud-based systems world.

It’s not good enough to focus on software, protocols and APIs. We have to begin to work together as an ecosystem to overcome the human and institutional barriers to better IT interoperability. Unfortunately, lack of interoperability often benefits software vendors, and as Hoff noted above, the only thing preventing this is greed.

Photo credit: Cory Doctorow

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• Quality of the cloud: best practices for ISVs
• A clouded view of Google Music