I’m a huge fan of cloud computing in general and AWS in particular. But I’ve long had an instinct that the economics of the choice between self-hosted and cloud provider had more texture to it than the patently attractive sounding “10 cents an hour,” particularly as a function of demand distribution. As a case in point, Zynga has made it known that for economic reasons, they now use their own infrastructure for baseline loads and use Amazon for peaks and variable loads surrounding new game introductions.

An analysis of the load profiles

To tease out a more nuanced view of the economics, I’ve built a detailed Excel model that analyzes the relative costs and sensitivities of AWS versus self-hosted in the context of different load profiles. By “load profiles,” I mean the distribution of demand over the day/month as well as relative needs for bandwidth versus compute resources. The load profile is the key factor influencing the economic choice because it determines what resources are required and how heavily these resources are utilized.

The model provides a simple way to analyze various load profiles and allows one to skew the load between bandwidth-heavy, compute-heavy or any combination. In addition, the model presents the cost of operating 100 percent on AWS, 100 percent self-hosted as well as all hybrid mixes in between.

In a subsequent post, I will share the model and describe how you can use it for scenarios of interest to you. But for this post, I will outline some of the conclusions that I’ve derived from looking at many different scenarios. In most cases, the analysis illustrates why intuition is right (for example, that a highly variable compute load is a slam dunk for AWS). In other cases, certain high-sensitivity factors become evident and drive the economic answer. There are also cases where a hybrid infrastructure is at least worthy of consideration.

To frame an example analysis, here is the daily distribution of a typical Internet application. In the model, traffic distribution is an input from which bandwidth requirements are computed. The distribution over the day reflects the behavior of the user base (in this case, one with a high U.S. business-hour activity peak). Computing load is assumed to follow traffic according to a linear relationship, i.e. higher traffic implies higher compute load.

Note that while labor costs are included in the model, I am leaving them out of this example for simplicity. Because labor is a mostly fixed cost for each alternative, it will tend not to impact the relative comparison of the two alternatives. Rather, it will impact where the actual break-even point lies. If you use the model to examine your own situation, then of course I would recommend including the labor costs on each side.

For this example, to compute costs for Amazon, I have assumed Standard Extra Large instances and ELB load balancer for the Northern California region. The model computes the number of instances required for each hour of the day. Whenever the economics dictate it, the model applies as many AWS Reserved Instances (capacity contracts with lower variable costs) as justified and fills in with on-demand instances as required. Charges for data are computed according to the progressive pricing schedule that Amazon publishes. To compute costs for self-hosting, I assume co-location with the peak number of Std-XL-equivalent servers required, each loaded to no more than 80 percent of capacity. The costs of hardware are amortized over 36 months. Power is assumed to be included with rackspace fees. Bandwidth is assumed to be obtained on a 95th percentile price basis.

Now let’s look at a sensitivity analysis. Notice in the above example, that a bit more than half of the total cost for each alternative is for bandwidth/data transfer charges ($35,144 for self-hosted at $8/Mbps and $36,900 for AWS). This is important because while Amazon pricing is fixed and published, 95th percentile pricing is highly variable and competitive

The chart above shows total costs as a function of co-location bandwidth pricing. AWS costs are independent of this and thus flat. What this chart shows is that self-hosting costs less for any bandwidth pricing under about $9.50 per Mbps/Month. And if you can negotiate a price as low as $4, you’d be saving more than 40 percent to self-host. I’ll leave discussion of the hybrid to another post.

This should provide a bit of a feel for how I’ve been conducting these analyses. Above is a visual summary of how different scenarios tend to shake out. The intuitive conclusion that the more spiky the load, the better the economics of the AWS on-demand solution is confirmed. And similarly, the flatter or less variable the load distribution, the more self-hosting appears to make sense. And if you’ve got a situation that uses a lot of bandwidth, you need to look more closely at potential self-hosted savings that could be feasible with negotiated bandwidth reductions.

Charlie Oppenheimer is a serial-CEO and currently an executive-in-residence at venture-capital firm Matrix Partners. His most recent company, Digital Fountain, was acquired by Qualcomm, and his previous company, Aptivia, was acquired by Yahoo. He blogs at stratamotion.com. 

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q2: Google closes the quarter with a bang
• From car to cloud: the future of the in-vehicle app landscape
• Finding the Value in Social Media Data

One of the areas we’ve seen huge demand for research from both our online and corporate customers is in the area of enterprise IT and cloud computing. From our tablestakes infrastructure, platform and storage coverage, to surging interest in all things big data, to evergreen topics such as enterprise mobility and social enterprise, we can’t produce research fast enough for our customers in this space.

So as 2012 will no doubt bring new opportunities, it will also bring a new face to GigaOM in Jo Maitland, our new Research Director for Enterprise at GigaOM Pro.  Many of you know Jo from her time as as an Executive Editor at TechTarget, where she steered the coverage of all things cloud computing, managing five dedicated cloud sites by the time she came to GigaOM.  In addition to her time at TechTarget, Jo cut her teeth as an industry analyst with Forrester and the 451 Group.

We have big plans in 2012 and expect Jo to be a big part of our success, please help me welcome her to GigaOM Pro.  You can follow her over Twitter and  over at GigaOM Pro, where she’s already finished her first report and will also be our infrastructure curator.

The first report of this came in late September, when marketing software service SEOMoz reported huge price spikes on the spot market. A sudden spike in the price of “m2.2xlarge” servers (normally $.44/hour) drove the price briefly up to $999/hour, causing a site-wise outage. While this was bad news for SEOMoz, it was probably worse news for the unlucky customers who ended up paying $999 for one hour of compute time!

Will you pay $999 per hour for a server?

Why would anyone bid such a high price? It’s hard to say for sure, but the unlucky winner of the auction probably did not expect to pay $999 per hour for a server. On the Amazon marketplace, your bid represents the maximum amount that you are willing to pay: you usually end up paying much less than your bid. Many buyers seem to have assumed that the price would never rise above the fixed-price “on-demand” rate charged by Amazon.

Unfortunately, it seems like a large number of people were using that flawed strategy. And when something changed in the spot market (perhaps a reduction in the number of machines available to rent, due to increased demand) the unrealistically high bids that customers made went into effect. Amazon has since posted a video describing the different strategies buyers on the spot market use. Litmus, one of the companies mentioned in the video, describes their strategy as “bidding high for convenience.” The $999 bidder who cornered the spot market on large servers was probably using an extreme version of this strategy.

My company (SlideShare) was also effected by the recent price spikes on the spot market. Several times in October and November, all of our EC2 servers disappeared at once because of a price spike (this had never happened before). Fortunately, the software code that manages SlideShare’s cloud servers responded automatically by renting new machines at the “on-demand” rate, so we didn’t experience any actual downtime, only degraded service. But after this happened to us several times, we have changed the mix of machines that we use so that only half of them are from the spot market, and the rest are on-demand.

Spikes are a recent problem

Looking through the pricing history for various classes of machines, it’s clear that these spikes are new, and that they are happening across almost all instance types, at least for servers that are on the East Coast of the United Sates. For example, “small” servers on AWS both spiked as high as $100 an hour twice in November, when the on-demand price for those servers is $.085/hr. “m1.large” machines also spiked as high as $40 an hour. Almost every class of servers has hit spikes of more than 10 times their retail price in last few months. What is going on?

It’s hard to say why the spot market is suddenly showing more price spikes. A drop in supply (from Amazon requisitioning machines for its own purposes or for renting in the on-demand market) or a spike in demand (from the Christmas e-commerce rush) could be to blame. It’s important to remember that the AWS spot market is not a typical market, with many buyers and sellers doing business over a neutral exchange. One seller is servicing many buyers, and is also operating the exchange.

Amazon benefits from customer anxiety about getting access to spot servers: they sell on-demand instances for a higher price, and pre-paid reserved instances for better cash flow. So it’s unrealistic to expect Amazon to do anything to “fix” these price spikes. From Amazon’s perspective, they are a feature, not a bug.

How to deal with EC2 spot price spikes

For customers of the AWS spot market, there are some best practices to be learned from these recent price spikes:

• Never EVER bid more than you are willing to pay for a server on the spot market. This is the most important lesson. Don’t even bother doing “convenience bidding” of double or triple the on-demand price: when the price starts to spike it will easily go way beyond any rational price. Do you want to be the gal who explains to the CEO why the company is paying $100 an hour for servers?
• Don’t run all your infrastructure on spot market machines. In fact, don’t run more infrastructure than you are prepared to lose on spot machines. We use a thumb rule of 50 percent at SlideShare, since our system can easily survive 50 percent of our machines disappearing at one time (which is what will happen during a price spike).
• Write the code that manages your cloud infrastructure so that it responds intelligently to spot market price spikes. If you can’t get a spot machine at a reasonable price, your code should automatically request an on-demand server.
• Consider having some “reserved instances,” so that you are guaranteed the right to a minimum base level of machines. I’ve argued in the past that reserved instances don’t make sense for startups, but it’s clear that when supply dries up at Amazon it happens all at once, without warning. Your portfolio of servers on Amazon is almost like a financial porfolio. You want some diversification between risky high-reward elements (spot market) and more conservative elements (reserved instances).

These are early days for real-time pricing of cloud computing, and the spot market on Amazon is finally acting like a real market, with extreme price fluctuations. The “free ride” of getting reliable spot priced machines for less than the on-demand price is over. So if you want to play with cheap cloud servers, make sure you have the infrastructure in place to handle a price spike that could make all your servers vanish in the blink of an eye!

Jonathan Boutelle is co-founder and chief technology officer of Slideshare a web site for presentations that relies heavily on cloud computing. Previously, Jonathan was a principal at Uzanto, (a UI consulting firm) and worked as a software engineer at CommerceOne (a B2B enterprise software firm) and Advanced Visual Systems (a 3D graphics startup) You can find his presentations on cloud computing at slideshare.net/jboutelle, and his Twitter is @jboutelle. He also blogs at www.jonathanboutelle.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• Quality of the cloud: best practices for ISVs
• For UK education, private clouds may make economic sense

The availability of cheaper yet powerful computing helps scientists crunch massive amounts of data that can lead to new discoveries. Last month, I spoke to a researcher at Johns Hopkins about this. Dr. Alex Szalay, of Johns Hopkins said:

“In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st-century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay said.

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing which will lead to new advances in astronomy and physics, much like the microscope’s creation in the 17th century led to advances in biology and chemistry.

This is why the search for exascale supercomputing matters. But compute in isolation isn’t enough. The other elements that are coming together are faster networks to move data around (although they aren’t fast enough) and a more collaborative culture in science and technology across geographies and disciplines. Broadband helps with that of course. The use of open-source software in scientific computing and of cheaper ways to harness massive compute power, either though the cloud or through GPUs, also allow more people to play with machines that are akin to the top-of-the-line supercomputers of 5-6 years ago.

So the democratization of compute, data analytics, the data itself and fast networks are changing how deeply scientists can look at a problem. And it offers a wider lens to put research in context thanks to collaboration and interdisciplinary studies. Today it’s a new form of chemical bonding, but next it could be a revolution in energy production inspired by the new bond. And with these new tools, discoveries will happen faster and be applied in more places. That’s cool even if chemistry isn’t your thing.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Finding the Value in Social Media Data
• Amazon’s DynamoDB: rattling the cloud market
• Cloud computing 2012: a pessimist’s guide

Joe Weinman at GigaOM Structure 2011

The public Internet and the cloud shouldn’t mix, according to a paper out today from Joe Weinman of HP. Cisco seems to agree, if Tuesday’s announcement of its CloudVerse suite of products is any indication. A growing number of endpoints, the multiple services built within web applications, and the infinite variety of demands made on any web-based service mean the network can’t be trusted to run over the top.

The network is the cloud, so it needs to be agile, smart and billed based on usage.

Instead, the industry will need to move to pay-per-use, dynamic networks where possible to improve the economic benefits of cloud scenarios and deliver defined quality-of-service for applications that will require low latency, argues Weinman. Weinman, who moved over to HP from AT&T last year, is a deep thinker on the economics of cloud computing. He also argues that bandwidth will eventually be charged on a pay-per-use model for both consumers and enterprises.

He makes a good case for the importance of a smarter network in the context of delivering cloud services, something Cisco’s CloudVerse announcement Tuesday also supports. CloudVerse basically organizes Cisco’s existing networking products for the data center and links them back to the networking gear already in carrier and service provider networks, with the idea being that an intelligent network can take the fuzziness out of managing applications in the cloud.

Complex apps and infinite endpoints make quality of service more important.

It’s true that applications are growing more complex and relying on more protocols to deliver a variety of services over the web. Take, for example, an application like Google+. There are real-time streaming elements, a video conferencing set-up and document sharing. Each different element requires different levels of network quality, which is why Weinman argues for networks that run faster, not just on a megabit-per-second basis, but also with less latency. From the paper:

Human performance studies show that 200 to 250 milliseconds is acceptable for multimedia conferencing and collaboration applications. However, interactive tasks such as keystrokes and mousedowns must be responded to within about 150 milliseconds [10], and emerging online games require even lower latencies.
There are also empirical results showing the importance of low latency not just in terms of user experience, but in terms of revenue. Lower latency directly correlates with increased revenue [11].

Add in the complexity at the end point in terms of the number of devices that connect to the network and it gets worse. Sensor networks, plus more devices per person and more concurrent streams coming in per device (as in personal video recorders) require more bandwidth. It also requires more intelligent bandwidth that can allocate resources and deal with emergent effects like in-office or in-home congestion and odd spikes in traffic in case of unexpected events. For example, a pipe breaking in a sensor-equipped home in the middle of the afternoon when the house is empty may create a sudden spike in traffic as humidity sensors activate, power gets shut in certain areas and you check in via a home camera system to see why your home network is going crazy. But because that’s an unexpected spike in a normally dull time, will your service provider have the bandwidth capacity to meet that event?

Of course, there’s something on cloudbursting and software-defined networks.

Weinman also offers the Holy Grail of true cloudbursting as an example where adding network intelligence makes it easier to scale a workload from one data center to another in times of peak demand. He lists five ways of doing this, beginning with the simplest idea of dividing up tasks between various clouds, which requires little to no network intelligence. He concludes with a network that can push a huge amount of data as needed and very quickly, but which would require infinite bandwidth. Since this last approach is impractical, he suggests providing pay-per-use bandwidth as the easiest way to instantly replicate data while keeping costs in line.

To help deliver the type of fine-grained control that intelligent networks will need, Weinman believes software-defined networks, such as those created using protocols like OpenFlow, are a way to add intelligence and flexibility. Using open protocols to create the networks are a good way to make sure that the added intelligence doesn’t act as a way to lock in users. Weinman also covers additional topics that will require research in bringing forth these new networks for cloud computing, and I highly recommend folks check out his paper.

This sounds great; so how do we co-opt it to sell products?

So what does this have to do with Cisco’s marketing effort around CloudVerse? Essentially, with the suite of products that wrap data center networking in with the networks of service providers for wireline and mobile broadband, Cisco is recognizing that a holistic, intelligent network could be a huge selling point for those concerned about piecing together their own fragmented network elements to deliver web services and cloud services. A quote from the Cisco release sums up the news nicely:

“Until now cloud technology resided in silos, making it harder to build and manage clouds, and to interconnect multiple clouds, posing critical challenges for many organizations,” said Padmasree Warrior, Cisco senior vice president of engineering and chief technology officer. “Cisco uniquely enables the world of many clouds – connecting people, communities and organizations with a business-class cloud user experience for the next-generation Internet.”

Cisco and Weinman are not alone. Alcatel-Lucent recently outlined its vision of as service provider cloud that adds intelligence to the network in a way that many enterprise and business customers will find appealing.

Could someone build a fully functioning network without resorting to all-Cisco gear, or perhaps even Weinman’s view of the intelligent network? Yes, but it takes skill and dedication that places such as Google, Yahoo, Facebook and other webscale operators have, and other companies just don’t seem to want to bother with.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q3: OpenStack and flash step into the spotlight
• Infrastructure Q2: Big data and PaaS gain more momentum
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4

There are companies trying to build better sensors, and those trying to make new ways of programming such sensor networks, but Axeda is trying to create the intelligence in the cloud to monitor and manage the sensors in real time. Axeda, a Foxboro, Mass.-based company that’s been around since 2000 and which has raised $16 million from JMI Equity and MMV Financial, has built a software platform for sensor networks.

Today, the platform handles more transactions per day than Twitter handles tweets, according to Joseph Biron, a senior director, product innovation at Axeda. When pressed, he said it was about 10,000 per second, although he expects that number to quadruple in the next year as more and more devices are connected. In March, Twitter said it saw almost 7,000 tweets per second at its highest point so far.

To handle this, Brion says Axeda has built its own NoSQL data store that he doesn’t want to disclose too much about. But he did say the Axeda engineering team follows Twitter, Facebook and other webscale businesses carefully to understand how they are handling their large amounts of data. With Ericsson predicting there will be 50 billion connected devices by 2020, he’s pretty sure the Axeda cloud will end up processing far more transactions that some of these household names.

And this assumes that not every sensor will be connected to a monitoring cloud. “Think about a building” Brion said. “There will be sensors on the fire panels and doors and windows. That’s thousands of them and they will likely connect and centralize through a gateway.” However, that’s still a lot of data coming in from one building, so Axeda hopes it is building a system that will be able to scale exponentially with the number of new devices added to its service as opposed to linearly.

He explains that in addition to monitoring, a dashboard, and the ability to send out updates or actions to the sensors, the Axeda software service will soon add more automated reactions. So, for example, if a building’s sensors determine the building is too hot, the information from myriad sources can be analyzed in the Axeda cloud and then the Axeda software can tell the building’s air conditioner to lower the temperature.

That can happen today, but the cloud component becomes more compelling when you bring in a third-party such as a utility, that can send the Axeda cloud a signal asking for power conservation, which can then push out that information to the building so it lowers the temperature in response. Because most sensor networks run on proprietary software, as opposed to something standardized or open source, Axeda has created an overlay in its service to translate the proprietary signals from devices manufactured by Honeywell, Emerson, Johnson Controls or others, into something that different systems can understand.

We’re not there yet, but this is the future that Axeda has decided to bank on. That’s why it has moved from tracking wireless assets to creating a cloud-based platform for managing all connected network devices. In addition to energy management, Brion thinks connected advertising, coordinated traffic management and other areas will be improved by a sensor network that’s controlled in the clouds.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• Dissecting the data: 5 issues for our digital future
• Connected world: the consumer technology revolution

Twitter has become an important information source for marketers, branding experts and others wanting to track or try to correct or capitalize on public perceptions of their companies based on Twitter’s “firehose” of information. All of that data, which often includes customer complaints or reviews of products and services, contributes a significant amount to the big data boom.

Twitter has authorized only two companies to sort through and track that Twitter data: DataSift and Gnip.

In a statement, DataSift founder Nick Halstead, who also founded TweetMeme, said:

Social media has amplified the already fast-paced nature of business today. Companies don’t have the luxury to sift through hundreds of millions of data streams every day, only to second guess the appropriate action. What they need is definitive access to real-time intelligence that is impactful to their business — allowing them to easily and quickly detect and respond to major trending events, social behaviors, customer preferences — and ultimately, avert any impending crises. We have been amazed with the demand for our platform in the U.S. and are opening an office to cater to this demand.

DataSift’s  users are not limited to keyword search: They can use complicated filters that take into account location, gender, sentiment, language and Klout influence. DataSift’s cloud-based services are available on a subscription basis and on demand.

Photo courtesy of Flickr user Remko van Dokkum

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Finding the Value in Social Media Data
• Connected world: the consumer technology revolution
• DataSift highlights more limitations in the public cloud

The original Cray supercomputer

In the past decade supercomputers were dressed-up versions of Intel’s x86 machines, but increasingly supercomputers are borrowing innovations (and silicon in the form of ARM-based chips or DSPs) from the mobile and big data realms to add speed without guzzling too much power.

Prior to this century many supercomputers really were a different animal entirely, sporting specialty chips and software. But the industry turned to commodity chips in the early 2000s. Now, to meet the demands of exascale computing at low power, chip makers are taking inspiration from the cloud computing and mobile industries.

ARM tries supercomputing on for size

As the Supercomputing 2011 show gets under way in Seattle, Nvidia, Texas Instruments, ARM and others are announcing new silicon to power the machines we rely on for science, climate prediction and high-end simulations in industries that range from oil production to car design.

Nvidia is a fairly recent newcomer to the supercomputing market, but it has made huge strides since 2008, when it first starting pushing its graphics processors (GPUs) as a way to boost speed while keeping energy usage in check. It said it would use its high-end GPUs and its new GPU-plus-ARM chip to build a new supercomputer in Spain. This is the first time an ARM-based processor has made its way into a supercomputer. ARM thus far has been the chip of choice inside cell phones and tablets.

Accelerator chips advance in supercomputers

Japan's K supercomputer is the fastest in the world.

Nvidia is doing well with its GPUs, given that in the top 500 ranking of the world’s fastest supercomputers, 39 systems use GPUs as accelerators and 35 of these use Nvidia chips. The graphics processors are used in supercomputers because they can handle massively parallel tasks that high-end computing requires while using less energy than the typical CPUs made by Intel and AMD. Nvidia and its GPUs made their first appearance on the list in 2008, and the last time the top 500 list was published, six months ago, Nvidia chips were in 17 machines. To go to 35 today is a pretty big uptake.

Perhaps inspired by Nvidia’s success in getting its GPUs onto supercomputers, Texas Instruments is bringing its digital signal processors to the mix for high-performance computing. DSP chips are really good at math, and they are used in telecommunications chips and in routers. TI has been thinking about this for a while, but Monday was its first launch into the market formally.

New chips for the cloud

The same power-efficiency issues that plague those trying to advance supercomputing are hitting those who run webscale applications, from Facebook to Amazon Web Services. And while the cloud and web-scale data center operators aren’t looking for specialty gear, like Infiniband for networking, they are running one or a few applications on their hardware, similar in some ways to a supercomputer, where all workloads are optimized for speed.

This is why certain chip and hardware companies, such as Tilera, Calxeda and Applied Micro, see an opportunity to redesign the silicon and gear inside the cloud. Meanwhile, companies such as Adapteva, which makes a massively multicore chip for cell phones and HPC, see an opportunity in pushing into supercomputers and mobile handsets, where the need for more-powerful processors and lower power consumption are always at war. And with ARM piggybacking on this trend thanks to Nvidia, it’s clear that supercomputers want to be super without the influence of PCs.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Overview, Q2 2010
• Infrastructure Q4: Big data gets bigger and SaaS startups shine

Businesses can — and do — use it to offload chunks of IT infrastructure as needed.

The Charleston Symphony Orchestra, for example, had to cut its IT staff but still had a server running email and other apps in a closet. Instead of keeping that thing chugging away, they went to hosted email and put the rest of that server data in — you guessed it — Dropbox’s cloud, he said.

The company’s success has been greatly aided by the fact that consumers want the same nifty tools they use from home, in the office, Houston told attendees of the GigaOM RoadMap Conference on Thursday afternoon.

People’s work tools used to be better than their tools at home, but that’s no longer the case. So if they use Dropbox for their family photos, chances are they’ll start using it for their work documents.

It also helps Dropbox’s case that today’s connected world isn’t always all that connected. “My mom has her music in iTunes but her phone is a Droid. So even though all of her devices are connected, they don’t talk to each other,” Houston said.

Clearly he sees Dropbox as the mother ship for all his mother’s (and everyone else’s) digital cargo. The proliferation of mobile devices has only driven more demand.

That cloud-based repository can also alleviate a lot of the headaches around moving big chunks of data around for business users or consumers. “Big email attachments and uploads — that’s the kind of thing that goes away if all of your stuff is in the cloud by default,” he said. “We can preview it, transcode it.”

As for what’s next, he didn’t provide a ton of detail, but he hinted that Dropbox will get better at uploading and managing all sorts of files.

“The way we manage files on a computer is insane. We’ve had this system for decades, but there’s still no one button that says ‘put this online,’” he said.

Dropbox will also be able to store not only a person’s photos but the metadata about that photo, the location information. “All of these things become possible. We can index all that metadata in the pictures and then tel you where the picture is taken, and maybe give you all the pictures taken within ten mile radius.” This sounds like a lot more than storage.

Dropbox is not alone in this hot cloud storage market: it competes with Box.net, Carbonite, Backblaze and others that are duking it out for the cloud storage needs of consumers and businesses. But it’s clearly growing gangbusters. The 45 million users Dropbox now claims is up from 25 million in April. The company just closed $250 million in funding.

Photo by Pinar Ozger.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• What Enterprise Software Vendors Could Learn from the Consumer Space
• NewNet Q4: Platform mania and social commerce shakeout
• Dissecting the data: 5 issues for our digital future

It's a frickin' laser, people!

Luxtera, which makes a optics chips that has characteristics of a standard silicon chip, has developed a hybrid chip for the data center market that can achieve speeds of more than 100 gigabits per second. Those are the same speeds that telecommunications firms are enabling via long-haul cables to handle the massive demand for bandwidth worldwide, but in this case are designed to handle the next wave of big data and networking-intensive applications inside webscale and cloud data centers.

Luxtera was founded in 2001 and builds chips that allow messages to be sent at the speed of light, but instead of using specialty materials that optics chipmakers such as Infinera use, Luxtera uses traditional silicon chips made using the CMOS process. This cuts down on the cost of the chips and makes it possible to use them for high-volume jobs, such as switching in the data center.

One cannot just add in an optical chip and expect it to work with the existing electronics on a board, so Luxtera created an integrated transceiver chip that combines transistor electronics with photonics on the same chip. This is cheaper, can be made in more traditional fabs and puts optical in reach of the data center operators. From the release:

Luxtera’s single chip opto-electronic transceiver includes four fully integrated 28Gbps transmit and receive channels powered from a single laser for an aggregate unencoded data rate of up to 112Gbps. The device is targeted for 100Gbps Ethernet, OTN and InfiniBand applications as well as emerging OIF (Optical Internetworking Forum) Short Reach (SR) and Very Short Reach (VSR) electrical interconnect to host systems. … The optical transceivers can be socketed directly onto the customers’ switch or server boards for both backplane and rack mount connectivity.

The story here is not just about a unique chip, but of hardware innovation racing to meet the needs of the next era in computing where the speed of transmitting information becomes as important as the speed of processing the information. Right now, there are bottlenecks on the chip and in the data center as the “brains” inside the servers wait for the information to get to them so they can process it. Intel and other companies are working on similar research into silicon photonics, but so far, Luxtera has announced the fastest transceiver so far.

The ascendancy of fiber isn’t just happening in our home broadband and long-haul networks, but also must occur inside the data center and even on the chips themselves as we demand more from our computers and networks. Luxtera’s chip helps usher in the age of light inside the data center in a way that doesn’t require the replacement of all the existing gear. Luxtera will sample the chips this year with the chips supporting both Ethernet and Infiniband applications.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Overview, Q2 2010
• Connected world: the consumer technology revolution
• Infrastructure Q3: OpenStack and flash step into the spotlight

Mu Dynamics makes a collection of products designed to show what happens when an app scales on infrastructure, from networks run by ISPs to servers running web apps in the clouds. As part of its efforts to sell more products, it’s embarking on a mission to educate both ISPs and app developers about how the software people are building today can affect the infrastructure if the software is widely adopted. Mu provides the equivalent of an infrastructure stress test.

For its latest test, Mu looked at Skype’s voice calling and chat functions across a bunch of different devices running different operating systems. A previous test looked at how Netflix traffic affected networks when compared to YouTube or other streaming video traffic. Netflix didn’t do well.

For the Skype test, Mu’s service made multiple calls to the Skype test number that lasted the same amount of time, measuring the data transmitted and the behavior of the packets to figure out how it would treat an operator, and how much of a data impact it would have on a consumer. Neither voice calls nor messaging had much of a data impact for the consumers, although in both cases running it on Mac OS X consumed the most data.

The results of this and other Mu Dynamics’ studies show how developers need to consider the infrastructure they’re running on as they design for the web. At the same time, the infrastructure providers have to understand the types of applications programmers are building and their consumers are consuming. In the case of server infrastructure, the end customers are the developers, which makes the relationship simple: the infrastructure or platform provider seeks to make developers’ lives easier so they will build on their services.

But in the case of ISPs, developers and the ISPs are both selling to end-consumers, and neither has a direct reason to talk to each other or work together to understand how the applications affect the infrastructure and vice versa. In a competitive broadband market, this might not matter because consumers could switch their ISPs easily if their Netflix or Skype ran poorly. However, because broadband access is essentially a duopoly and ISPs may have an economic interest in making sure certain apps run poorly, maybe Mu Dynamics’ can help provide — if not an incentive — at least information that could help booth sides meet in the middle.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected Consumer Q2: Digital music meets the cloud; e-book growth explodes
• NewNet Q2: Google closes the quarter with a bang
• From car to cloud: the future of the in-vehicle app landscape

The Open Compute battery cabinet.

The Facebook-led Open Compute Project launched a foundation Thursday to help it push the standardization of data center server hardware for webscale deployments. But as the project evolves it’s still hard to see where Facebook ends and Open Compute begins.

Leave the chassis to Open Compute and build something new.

The goal of the new Open Compute Foundation is to bring more vendors and voices into the mix, make sure their contributed intellectual property is well cared for, and to foster the idea that open-source development — so important in software — can benefit the stodgy world of data center servers. At the  Open Compute Project (OCP) launch in April, Facebook laid out building blocks for standard server designs. The idea is that other companies could build and innovate atop those designs and not waste time sweating the nuts and bolts.

“The main thing we want to achieve is accelerating the pace of innovation for scale computing environments and by open sourcing some of the base elements we will enable the industry in general to stop spending redundant brain cycles on things like re-inventing the chassis over and over and over and focus more on innovation,” Frankovsky said in an interview in advance of the foundation announcement. The effort will turn the data center, systems level and server hardware into commodity components designed for scaled out architectures.

The group has big backers with foundation directors including Silicon Valley superstar Andy Bechtolsheim who co-founded  Sun Microsystems and is now chief development officer of Arista Networks. Also on the board are Don Duet, head of global technology infrastructure for  Goldman Sachs;  Mark Roienigk, the COO of Rackspace; and Jason Waxman, general manager of Intel’s data center group. Frank Frankovsky, Facebook’s director of hardware and supply chain, is executive director.

What’s inside Open Compute today and planned for tomorrow.

Along with the creation of the foundation, Facebook announced the Open Rack 1.0 specification, which lays out the basic design for power distribution and cooling for the server rack. That spec will evolve over time, integrating such perks as rack-level power capping, and I/O on the backplane at some point, Frankovsky said.

Also on Thursday ASUS said it will open-source its motherboard designs and Mellanox plans to release specifications for 10 Gigabit Ethernet cards. So far the OCP effort has received intellectual property contributions from Red Hat– which will certify OCP servers. Other contributions came from AMD, Dell, and Cloudera. Arista Networks is also now an official member of OCP, although has no specific contributions to announce at this time.

The OCP  has also moved to make OCP hardware more broadly available, working with Synnex, a computer distributor and its manufacturing arm, Hyve, which will act as a hardware OEM. Silicon Mechanics, a maker of rack-mount servers, is also aboard. When the effort launched in April Dell and Hewlett-Packard  both showed off servers that incorporated some of the elements of Open Compute.

 Open Compute Foundation, born of Facebook, still pretty close

The Open Compute chassis.

The fact that a Facebook executive doubles as the foundation’s executive director is bound to raise some eyebrows if OCP wants to shake the perception that it is an effort directed by the social networking giant.  Other open-source projects, notably the Eclipse effort around Java development environments, really hit their stride only after the lead vendor relinquished control. (In Eclipse’s case, that was IBM.) More recently, Rackspace eased some concerns among the OpenStack software crowd by forming an OpenStack Foundation, and vowing to step back.

“We modeled this as closely as possible on the Apache Foundation. Each project starts at an incubation committee which names a lead and [is eventually] voted in or out as a project,” Frankovsky said. “I have one-fifth vote. If the others don’t think it’s cool, it’s not in.”

Frankovsky said the effort is well-funded for now through voluntary seed contributions but the funding model remains a work in process.

What’s next for OCP?

As for what’s next? Frankovsky said the first round of motherboards were based on Intel’s Westmere chip technology while version two will be based on Intel’s Sandy Bridge technology. “Intel and Hyve will do a fast-ramp program,” he said.  OCP has worked to get early access to Sandy Bridge technologies that would otherwise not be available until the second quarter.

Facebook itself is working on some storage specifications it would like to  talk about for its next round of contributions. “Storing data at this scale has some unique challenges. We’ll work on those contributions and with the rest of the community on this,” he said.

The OCP remains focused on the compute platform itself, although Frankovsky didn’t rule out possible future forays into other parts of the data center universe.

Asked if networking was on the agenda, he said:  ”Andy Bechtolsheim has a lot of interest in networking but for now we’ve excluded networking from Open Compute. There’s already ONF [the Open Networking Foundation] and we don’t want to compete, but if the community thinks we should look at the physical layer of Open Compute, that’s a possibility.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4

IBM’s outgoing CEO, Sam Palmisano, divested some of the thinner-margin stuff like PCs to focus more on the richer realms of software and services. Rometty worked with Palmisano on this services-fed strategy for some time, and if it’s succeeding, continuity is the best policy, said several IBM watchers.

“Why should she change the strategy when she helped build the strategy and it’s working so well?” asked Bob Djurdjevic, the president of Annex Research, who has followed IBM for more than 20 years.

“Cloud services have been a distinguishing characteristic of IBM for years and it’s been interesting to watch other companies poo-poo this strategy — companies like Oracle, like Microsoft, like HP, who have since announced their own cloud strategies,” he added.

Indeed, Oracle CEO Larry Ellison used to scoff that IBM offered so much hardware and software just so customers would have to write checks to IBM services to integrate them. But once Oracle bought Sun Microsystems, he started holding up the IBM of another era as the ideal prototype of a company. Oracle unveiled its cloud computing earlier this month at its OpenWorld conference.

Judith Hurwitz, the president of Hurwitz and Associates, concurred that Rometty will stay the course. Rometty has been transitioning to this role for some time, Hurwitz said.

“Naming her CEO is the board endorsing [both] the cloud strategy and [Palmisano and Rometty's] policy of bringing the threads of IBM together in a very flexible way,” Hurwitz added. “The usual problem with these big companies is they have all these little fiefdoms fighting each other rather than working together.”

In 2002, Rometty helped engineer IBM’s $3.5 billion buyout of PricewaterhouseCoopers‘ consulting business before going on to lead IBM’s global sales effort. IBM now derives a big chunk of its revenue from both professional services (via IBM Global Business Services) and infrastructure/outsourcing services (IBM Technology Services). In its most recent quarter, IBM logged $10.3 billion from Global Technology, up 9 percent from the year-ago period, not figuring in currency changes. Global Business Services were up 6 percent, to $4.8 billion. IBM also said its cloud revenue for the first three quarters of this year was double its cloud revenue for the full previous year.

Rometty started at IBM in 1981 as a systems engineer, so she’s done her time in the tech trenches as well. She is known to be a very good communicator — both one-on-one and before large audiences.

“I give Palmisano credit. He picked someone who is more charismatic than he is. In fact, more charismatic than any tech CEO I’ve seen,” said Djurdjevic.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• The Case for Increased M&A in 2011: Actions and Outlooks
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

At the top of the shopping list is more security, more management and more data analytics know-how, Pat Gelsinger, president of Information and Infrastructure Products for the company, told reporters today at an event at Gillette Stadium in Foxborough, Mass. As big as EMC’s acquisitions of companies like RSA Security and Greenplum were in security and analytics respectively, Gelsinger clearly feels there’s more ground to cover.

“In security, with our acquisitions of Archer and NetWitness, we know there’s still a lot of work to be done in this new virtualized cloud world. We don’t think it’s finished yet. While we will do organic innovation in that area, we see a lot of companies and different ways to approach authentication and managing that authentication across multiple mobile devices,” he said.

And, with the proliferation of PaaS, SaaS and IaaS providers, there needs to be a way to manage the relationships between those providers. “We don’t have all the stuff to do that,” he said.

Customers need the same sort of management and monitoring niceties that grew up with legacy enterprise hardware and a lot of that is just not there yet for the cloud.

“There’s a mature environment around mainframes with ITIL and CMDBs: That stack has not been built out for the virtualized world,” Gelsinger said. (ITIL stands for the Information Technology Infrastructure Library: a set of best practices for managing IT services; a CMDB, or configuration management database, is a repository that stores information about the pieces of an IT system.) For businesses to get to cloud-scale management, those are the sorts of things that need to be in place, he said.

The analytics market is so huge – Gelsinger put it at $70 billion and growing at 14 percent to 20 percent per year — there’s been a big acquisition binge around it with IBM, EMC, Oracle and SAP snapping up analytics or business intelligence companies. “There’s a variety of visualization, data transport tools, tools in the ETL (extract, transform, load) arena that have to re-emerge in this big data domain,” Gelsinger said.

As for its core storage business, EMC is more likely to grow organically rather than acquisition. “We think we’re pretty good there post-Isilon. In our storage family, we think we have the assets we need; one area of interest is flash and we’re doing that organically with FAST [EMC's Fully Automated Storage Tiering software] and Project Lightning development.” EMC bought Isilon Systems and its scale-out file system last year.

Project Lightning, a server-side, solid-state flash drive that EMC is developing with an unnamed partner, is the company’s response to Fusion-io’s  flash storage. EMC is revising its FAST software so high-priority data will be handled by the fast flash storage while less urgent data is funneled off to cheaper (and slower)  SAS or SATA drives.

EMC clearly has the wherewithal to build or buy its way into many arenas. As GigaOM’s Derrick Harris reported this morning, EMC CEO Joe Tucci said Wednesday night that the company will spend about $2.4 billion (or 12 percent of its projected revenue) on R&D this year. And, from 2003 though 2010, it spent $10.5 billion on R&D and $14 billion on acquisitions.

Photo courtesy of Flickr user SiliconANGLE PHOTO

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q3: OpenStack and flash step into the spotlight
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q2: Big data and PaaS gain more momentum

Stuart Elby, VP and network architecture & technology chief technologist for Verizon Digital Media Services, laid out how the promise of software-defined networking could make the company’s cost curve match its revenue by cutting down on the need for expensive gear that is costly to buy and even more costly to operate. In a conversation before his presentation, Elby explained how Verizon’s network can view every single packet on the network, but how keeping track of those packets is both a big data problem and expensive from a network management perspective.

For a while already, Verizon has been trying to host as much of its network as possible on commodity boxes, running commodity servers and some Sun boxes at the core of its network and keeping the big, expensive gear from the likes of Juniper and Cisco at the edge of the network. Elby was cagey about how his preference for lower-cost bit delivery might affect Verizon’s big suppliers, especially since he was seated onstage next to David Ward, CTO of Juniper. When Elby said he wasn’t planning on ditching any of the recent Juniper boxes he’s just bought, Ward quipped, “I appreciate that, Stu.”

However, Elby is dealing with his costs exceeding his revenue, which he explained was coming in “a matter of years,” although he didn’t specify how many. The chart above is not limited to Verizon. Nick Mckeowan, an ONF board member and a pioneer of the protocol, said that he has seen charts like that one from other carriers, including Deutsche Telecom. It’s also a topic we at GigaOM have covered many times in the past.

So how exactly does a new protocol help?

OpenFlow is a protocol that allows someone to separate the intelligence inside a switch and router from the hardware itself. The promise of OpenFlow is that operators can create software-defined networks that are programmable. For more on the topic, see this video explaining it or this GigaOM Pro article (sub req’d). For an operator, it could make it far easier to direct the packets flowing around their networks because they could have more freedom and flexibility in programming their networks to do whatever the operator wanted.

Elby, for example, proposed several use cases, including traffic steering, which involves understanding what service a packet represented and what the subscriber’s plan was and then shunting that traffic over to the most appropriate path. This might mean recognizing that a succession of packets coming from Netflix is a streaming movie, so it could be sent on its way without further investigation, or it could eventually be a way to manage heterogeneous networks.

Another example comes form the data center world, which Verizon doubled down on when it bought Terremark this year for $1.4 billion. Elby describes a scenario where a customes trying to send huge files from one data center to another could guarantee their delivery by upping their bandwidth capacity on demand as opposed to paying for a consistent connection. In this way, Verizon begins to deliver capacity as a service in a manner similar to how Amazon delivers compute as a service.

Elby described more options, but the message underlying his talk was that OpenFlow and software-defined networks could lower Verizon’s costs, but it also turns Verizon into a service provider with a change in the type of cost model it will have. Understanding the technology as Verizon implements it and how it will change its spending on equipment and operating its network is one thing. Understanding the new business models that Verizon can implement as it provides what can become a multi-tenant, shared network model even for enterprise clients is another. It’s going to be fun, but I’m not sure if it will actually result in a lower mobile phone bill.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q2: Big data and PaaS gain more momentum
• The Structure 50: The Top 50 Cloud Innovators

Google Translate conversation mode, which debuted in January, is still in alpha and is in the process of getting better as it gets fed more speech data. It allows a user to speak their language into the Google Translate app and have their words spoken out loud in another language. Users can trade turns speaking into the app and have translated word spoken and rendered in text. With the addition of a dozen new languages, it should greatly expand the number of people who use the tool. Conversation mode is one of those tools that really show off the power of the cloud on mobile devices.

The service is also getting some additional features that help facilitate conversations. Users will now be able to see if the app has incorrectly recognized spoken words and correct them before they’re translated. They can add unrecognized words to their own personal dictionary. And users can also pinch to zoom or tap on a magnifying glass to view translations in full screen mode. This is helpful if you just want to show someone translated text instead of using conversation mode. The Translate App can handle text translations of 63 languages.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Flash analysis: the tech startup investment environment, Q3 2011
• Connected Consumer Q2: Digital music meets the cloud; e-book growth explodes