I sat down with EMC SVP and CTO John Roese on Tuesday at the company’s annual EMC World conference to find out what’s up. Here’s what he had to say.

On three companies under one roof

While they’re technically three separate companies, EMC is really in control. It’s the majority shareholder in VMware and owns more than 60 percent of Pivotal, its new joint venture with VMware that includes the Greenplum, Pivotal Labs, SpringSource, Cloud Foundry and Cetas business lines. When it comes to everyone working toward a common goal, Roese said, “The good news is that while there is independence, Joe Tucci is the chairman of all these companies.”

Roese calls himself the “gravitational center” of the three companies when it comes to technology. This is a reinvention of the CTO role at EMC, which used to be more of a research position. Now, he puts the stake in the ground and generally directs everyone toward it, even if they’re not all taking the same path to get there.

On why Pivotal happened and why it matters

My takeaway from Roese’s comments on formation of Pivotal is that Greenplum is really the linchpin of the whole company. At its core, Pivotal is about building big data infrastructure that can handle next-generation workloads, but it’s aware that broad adoption is only possible if that high technology becomes easier to consume. That means new higher-level applications, which is where SpringSource, Cloud Foundry and Pivotal Labs come into play.

All of this technically could have been accomplished by just selling Greenplum and Pivotal Labs (the only assets of the new company that was under the EMC umbrella) to VMware, but Roese said VMware wasn’t the right home because VMware is not so important in the places where next-generation workloads are popping up. There’s not a lot of VMware inside carriers’ data centers, he acknowledged, but there is a lot of OpenStack popping up. And there’s a lot of Amazon Web Services everywhere you look.

“We would like the big data infrastructure to not care about that,” Roese explained. From EMC’s perspective, it doesn’t need to own the middle — the cloud operating system, if you will — if it can still engage customers at the storage and application-platform layers.

On keeping independent while working an ‘unfair advantage’

Roese doesn’t think a vertically integrated approach is the best way to do business in today’s technology world, which is why EMC, VMware and Pivotal all operate independently and no one relies on another in order to work within customers’ data centers. That’s why VMware has its own cloud computing efforts but Pivotal is cloud-agnostic, why EMC storage can operate with any higher-level software and why VMware doesn’t care about what’s running underneath or, usually, above it.

However, he added, it’s only natural the three companies seek an “unfair advantage” from the incestuous bonds they share. What he means, of course, is that they should keep a close eye on what the others are doing and work together to ensure they’re all optimized for the same types of workloads. For example, Roese said, if EMC didn’t reconsider how storage had to perform given that virtualization is the norm or that technology like Hadoop exists, it would “become suboptimal or generic.”

The same holds true for Pivotal and VMware. Pivotal needs to think about how big data applications run on virtualized resources differently than on big bare metal systems, as well as on flash-based arrays like what EMC is about to roll out based on its XtremIO acquisition. VMware and EMC need to think about how their software-defined data center and software-defined storage approaches can build off each other.

From EMC’s perspective, it’s easy to see why this all matters. It is at its core an information infrastructure company, but “the challenging thing with that is that it’s a moving target,” Roese said. A company like EMC can’t get by on storage arrays alone anymore, but it also can’t be dumb enough to think it can be everything to everyone and still be good at anything.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data
• Cloud and data third-quarter 2012

That the new release of Ceph, dubbed Cuttlefish, focuses on Red Hat is interesting since Red Hat bought Gluster for its scale-out storage capabilities in 2011 and declared Gluster to be “OpenStack Ready” last month.

The consensus at OpenStack Summit was that Ceph has advanced faster than the Swift storage module that came out of Rackspace and which handles object storage only. But the promised appeal of OpenStack is that users can swap in and out compliant plug-ins as needed for different functionality.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Cloud computing 2013: how to navigate without a map

Researchers at the IT giant’s Almaden Lab in San Jose, Calif.  worked for ten days moving 10,000  individual atoms around on a microscopic surface to build the images of a boy and his interactions. It takes some massive gear to move 10,000 tiny atoms around and IBM’s lab had just the thing, a 2-ton scanning tunnelling microscope (STM).

One goal, according IBM, is to inspire kids to study and pursue careers in science and technology.  And, the work could lead to breakthroughs in storage and other technology fields.

According to an IBM blog post on the project:

“The ability to move single atoms, one of the smallest particles of any element in the universe, is crucial to IBM’s research in the field of atomic-scale memory. In 2012, IBM scientists announced the creation of the world’s smallest magnetic memory bit, made of just 12 atoms. This breakthrough could transform computing by providing the world with devices that have access to unprecedented levels of data storage. But even nanophysicists need to have a little fun. In that spirit, the scientists moved atoms by using their scanning tunneling microscope to make … a movie, which has been verified by Guinness World Records™ as The World’s Smallest Stop-Motion Film.”

Enough chit chat though, check out the film for yourself:

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• What Enterprise Software Vendors Could Learn from the Consumer Space

Like a bad episode of Hoarders, people love to store all things digital, most of which will never be accessed again. And, like a bad episode of Storage Wars, our love of storing crap means we need more places to store it. Today’s content has outgrown even the hydro-electric dam powered Mega Data Centers built just yesteryear. Increasingly, operators are turning to distributing their information across multiple geographically dispersed data centers. As the number, size, and distances between the data centers have steadily grown, timing distribution and accuracy has likewise grown in importance in keeping the data centers in sync.

In a previous article I discussed new standards being developed to increase the accuracy of timing for the internet and other IP-based networks. Current systems and protocols offer milliseconds of accuracy. But that just isn’t enough as we depend more on real-time information and compute, storage and communications networks become more distributed. While people often cite the importance of timing on mobile backhaul networks in the next-genration LTE-Advanced networks,there has been less publicity around the need for these new timing technologies in the continued growth of data centers.

The rise of Hadoop in an age of digital garbage

Massive storage of data appears to occur in periods, very analogous to dinosaur evolution. A database architecture will rise to the forefront, based upon its advantages, until it scales to the breaking point and is completely superseded by a new architecture. At first, databases were simply serial listed values with row/column arrangements. Database technology leapt forward and became a self-sufficient business with the advent of relational databases. It appeared for a while relational databases would be the end word in information storage, but then came Web 2.0, social media, and the cloud. Enter Hadoop.

A centralized database works, as the name suggests, by having all the data located in a single indexed repository with massive computational power to run operations on it. But a centralized database cannot hope to scale to the size needed by today’s cloud apps. Even if it could, the time needed to perform a single lookup would be unbearable to an end user at a browser window.

Hadoop de-centralizes the storage and lookup, as well as computational power. There is no index, per se. Content is distributed across a wide array of servers, each with their own storage and CPU’s, and the location and relation of each piece of data mapped. When a lookup occurs, the map is read, and all the pieces of information are fetched and pieced together again. The main benefit of Hadoop is scalability. To grow a database (and computational power), you simply keep adding servers and growing your map.

Even Hadoop is buried under mounds of digital debris

It looked like Hadoop would reign supreme for generations to come, with extensions continuously breathing new life into the protocol. Yet, after only a decade, databases based upon Hadoop such as Facebook are at the breaking point. Global traffic is growing beyond exponential, and most of it is trash. Today’s databases look more like landfills than the great Jedi Archives. And recently hyped trends such as lifelogging suggest the problem will get much worse long before it gets better.

The main limitation of Hadoop is that it works great within the walls of a single massive data center, but is less than stellar once that database outgrows the walls of a single data center and has to be run across geographically separated databases. It turns out the main strength of Hadoop is also its Achilles heel. With no index to search, every piece of data must be sorted through, a difficult proposition once databases stretch across the globe. A piece of retrieved data might be stale by the time it reaches a requester, or mirrored copies of data might conflict with one another.

Enter an idea keep widely dispersed data centers in sync — Google True Time. To grossly oversimplify the concept, True Time API adds time attributes to data being stored, not just for expiration dating, but also so that all the geographically disparate data centers’ content can be time aligned. For database aficionados, this is sacrilegious, as all leading database protocols are specifically designed to ignore time to prevent conflicts and confusion. Google True Time completely turns the concept of data storage inside out.

Introducing Spanner

In True Time, knowing the accurate “age” of each piece of information, in other words where it falls on the timeline of data, allows data centers that may be 100ms apart to synchronize not just the values stored in memory locations, but the timeline of values in memory locations. In order for this to work, Google maintains an accurate “global wall-clock time” across their entire global Spanner network.

Transactions that write are time stamped and use strict two phase locking (S2PL) to manage access. The commit order is always the timestamp order. Both commit and timestamp orders respect global wall-clock time. This simple set of rules maintains coordination between databases all over the world.

However, there is an element of uncertainty introduced into each data field, the very reason that time has been shunned in database protocols since the dawn of the data itself.

Google calls this “network-induced uncertainty”, denoted with an epsilon, and actively monitors and tracks this metric. As of summer 2012, this value was running 10ms for 99.9 percent (3 nines) certainty. Google’s long term goal is to reduce this below 1ms. Accomplishing this will require a state of the art timing distribution network, leveraging the same technologies being developed and deployed for 4G LTE backhaul networks.

A modest proposal

While True Time was most likely developed to improve geographic load balancing, now that accurate time stamping of data exists, the possibilities are profound. The problems associated with large databases go beyond simply managing the data. The growth rate itself is unsustainable. Data storage providers must do more than grow their storage, they must also come up with ways to improve efficiencies and ebb the tsunami of waste that is common in the age of relatively free storage.

It’s a dangerous notion, one simply must challenge the basic tenet that all data is forever. Our minds don’t work that way, why should computers? We only hold on to key memories, and the further the time from an event, the fewer the details are held. Perhaps data storage could work similarly. Rather than delete a picture that hasn’t been accessed in a while, a search is performed for similar photos and then only one kept. And as time passes, perhaps rather than simple deletion, a photo is continuously compressed, with less information kept, until the photo memory fades into oblivion. Like that old Polaroid hung on the refrigerator door.

Jim Theodoras is director of technical marketing at ADVA Optical Networking, working on Optical+Ethernet transport products.

Dinosaur image courtesy of Flickr user Denise Chen.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• 9 Companies that Pushed the Infrastructure Discussion in 2010

As is the case with software-defined networks, the Jeda controller software will separate the control plane from from the data plane. It will run on a virtualized server and take charge of the intelligence that would otherwise reside on a switch. SDN can have a wide range of benefits, although it generally can permit more programmability and elasticity of networks. For Jeda, virtualizing storage-area networks could allow customers to generate and disable those networks in response to changes in demand.

Other companies other than Newport Beach, Calif.-based Jeda are likewise looking to make software-defined storage a reality, including Convergent.io, ScaleIO and SwiftStack. Approaches differ among those entities, but they share the goal of bringing storage up to the level of programmability of compute resources and, increasingly, networks.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The promise of SDNs in the enterprise
• Takeaways from the second quarter in cloud and data
• A near-term outlook for big data

I’m not certain what this exponential growth means. It probably means a lot of people are using Amazon Web Services. It probably means they’re building a lot of next-generation applications storing a whole lot of photos and other things (I’m thinking of Pinterest, Instagram, Dropbox, you name it). It probably also means big data is for real in the cloud, as Amazon S3 is the default storage system for AWS’s Elastic MapReduce Hadoop service.

Whatever it means, though, 2 trillion is a lot of anything and probably means Jeff Bezos was right to spend so much time talking AWS in discussing the company’s annual report. At an estimated $2 billion or so in revenue and storing probably a not insignificant amount of data for so many fast-growing web services, AWS looks to be around for the long haul.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• AWS Storage Gateway jolts cloud-storage ecosystem
• 9 Companies that Pushed the Infrastructure Discussion in 2010
• The fourth quarter of 2012 in cloud

However, as a press release explaining the new report highlighted, defining “storage” for the purposes of big data is an exercise in subjectivity. There are systems for archiving data, and systems for storing post-processed data and systems — like the Hadoop Distributed File System — that put storage on the same servers that process data. There also are storage systems designed for operational data and those designed for transactional data, and very likely something in between.

Presumably, these numbers don’t account for the amount of storage baked into analytic database appliances like those from Teradata and Netezza. And, although the report doesn’t appear to address it, there also will be a market for storing data in the cloud — both provider-side and user-side. Even here, there are a variety of options from Hadoop services to data warehouse services to software-as-a-service applications.

The storage research is just IDC’s latest attempt at quantifying a big data movement that spans a wide section of individual markets. On Monday, for example, the firm predicted the market for analytics services would reach $70.8 billion by 2016. In January, it said big-data-specific servers, storage, networking, software and services will create a $23.8 billion market by 2016.

As we’ve said time and time again, though, trying to put a dollar value on the big data market is in many cases akin to herding cats (that we might also shear for fur and that might provide a valuable service killing off crop-damaging varmints). There are so many disparate facets and delivery models that touch so many different business uses and revenue sources that it’s difficult to capture big data, or any of its individual components, into a single market. But however IDC and other research firms define it, the only thing that matters in the end is probably the ever-rising revenue arrow.

Feature image courtesy of Shutterstock user Mmaxer.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Health care and big data in 2012

EMC is aiming the new product at companies with more than 25 users. It can work with cloud-based storage and on-premise storage appliances such as EMC’s Isilon network-attached storage and Atmos object-based storage. That could make Syncplicity’s Enterprise Edition an appealing option for IT administrators who want to support the bring-your-own-device trend with a simple user interface but don’t want to worry about the security implications of using Dropbox and other offerings.

Companies that use EMC’s Documentum enterprise content-management software can also sync files in the Syncplicity Enterprise Edition. Syncplicity keeps a customer’s data centers in multiple geographies automatically updated, and users access files from the nearest data center to keep latency low.

Instead of requiring customers to pay a different amount of money each month to reflect elastic use of storage resources the way Amazon Web Services does for its Simple Storage Service (S3) and other products, Syncplicity customers pay based on the number of users — a more palatable option for larger businesses with hundreds or thousands of employees, said Jeetu Patel, Syncplicity’s general manager.

While EMC might have wanted to have a product for small businesses in its line when it bought Syncplicity last year, it’s not surprising to see the company create a version that makes sense for large businesses, too. EMC is no Box, much as Box wants to add enterprise customers. But now it comes closer, by crossing ease of use with the security advantages of on-premise deployments.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A near-term outlook for big data
• Dissecting the data: 5 issues for our digital future
• Report: The Future of Data Center Storage

According to Fusion-io, the deal will leave safe the open-source distribution of SCST. On the other hand, ID7 makes its money off helping those other hardware vendors implement SCST in their commercial products, so in that sense Fusion-io just bought its way closer to the center of the software-defined storage universe.

Here’s what Fusion-io product chief Gary Orenstein had to say in a statement on Monday:

“As with other kernel contributors across the Linux stack working at Fusion-io, we will continue to support the open philosophies that have made ID7 and SCST successful. In addition to maintaining an open source version of SCST, Fusion-io will continue to contribute to the open source distribution as we develop software solutions to help define the all-flash data center.”

Software-defined storage, where storage is abstracted in much the same way as has been done with servers, is a lucrative scene right now. Just in recent months, we’ve reported on major investments for players such as SwiftStack, ScaleIO, Convergent.io, Nutanix and Jeda Networks.

The supposed benefit of the software-defined storage approach is the ability to weave shared storage pools out of standard server-attached hardware, so usage can be quickly scaled up and down as needed. ID7 has already been working with Fusion-io on its ION Data Accelerator software, which does this in an interface-agnostic fashion, and will indeed be folded into this side of Fusion-io’s business.

Again from the statement, here’s ID7 co-founder Mark Klarzynski:

“We had an opportunity to work with Fusion-io on the development of the ION Data Accelerator when it became apparent that the team has been founded on a culture of architecting software innovation deep within the Linux operating system kernel to deliver significant breakthroughs in modern storage architectures. We’re excited to join the Fusion-io team of world class engineers and developers to work together on open, software defined solutions to today’s most challenging data demands.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Quality of the cloud: best practices for ISVs
• Is the Future of Enterprise Completely Open Source?
• Will Storage Go the Way of The Server?

Those optimistic about 2013 point to the second half of the year (now only four months away) as when we’ll begin to see some stronger growth. In their view, Europe will be better, the U.S. will be stronger, China will begin to reaccelerate in earnest, currency will be stable, and (perhaps most importantly) the year-over-year (YoY) comparisons between 2012 and 2013 will be easier because mid-2012 was so weak. Moreover, as EMC CEO Joe Tucci stated during the company’s recent earnings call, “You can’t starve IT for too long,” which suggests that better times must get rolling again. On the other hand, CEO John Chambers’ statement in the Cisco earnings call, that perhaps 2012 represents the new normal, might be a better indicator of what we can expect in 2013. We’ll see.

That said, here’s what we’re expecting for 2013.

Table 1. 2012 and 2013 worldwide IT spending (in $ billions)

Overall, smartphones and tablets will lead the way, with a 5.6 percent worldwide IT growth rate in 2013. While Apple’s tepid forward guidance in its recent earnings call and its implications for smartphones and tablets gives us pause, we expect the category to continue to lead, with growth in the 20.4 percent range in 2013. In hardware, 2013 will look like 2012, with smartphones and tablets buoying the otherwise-weak spending in the PC, peripheral, and server segments. (This will be offset somewhat by stronger storage and network hardware spending.) Software continues to flourish — especially solutions-related software like SaaS-enabled customer-relationship management (CRM), supply-chain management (SCM), and industry vertical systems like health care. This sector will continue to grow in 2013. Finally, services has experienced a punishing 2012, and this area will only look modestly better in 2013.

A methodological note: We’ve reevaluated our model and determined that we have underweighted business-process outsourcing (BPO) as a service category. As such, we are restating 2012 and 2013 growth rates to accommodate a higher weighting of BPO in our services spending. The net is that growth in services and all spending are a bit more robust, since BPO is growing faster than the aggregate of traditional IT services. The full methodology can be found at the end of this report.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• IT spending update, third quarter 2012
• IT spending update, fourth quarter 2012
• Work media tools in 2012 and beyond