The first report of this came in late September, when marketing software service SEOMoz reported huge price spikes on the spot market. A sudden spike in the price of “m2.2xlarge” servers (normally $.44/hour) drove the price briefly up to $999/hour, causing a site-wise outage. While this was bad news for SEOMoz, it was probably worse news for the unlucky customers who ended up paying $999 for one hour of compute time!

Will you pay $999 per hour for a server?

Why would anyone bid such a high price? It’s hard to say for sure, but the unlucky winner of the auction probably did not expect to pay $999 per hour for a server. On the Amazon marketplace, your bid represents the maximum amount that you are willing to pay: you usually end up paying much less than your bid. Many buyers seem to have assumed that the price would never rise above the fixed-price “on-demand” rate charged by Amazon.

Unfortunately, it seems like a large number of people were using that flawed strategy. And when something changed in the spot market (perhaps a reduction in the number of machines available to rent, due to increased demand) the unrealistically high bids that customers made went into effect. Amazon has since posted a video describing the different strategies buyers on the spot market use. Litmus, one of the companies mentioned in the video, describes their strategy as “bidding high for convenience.” The $999 bidder who cornered the spot market on large servers was probably using an extreme version of this strategy.

My company (SlideShare) was also effected by the recent price spikes on the spot market. Several times in October and November, all of our EC2 servers disappeared at once because of a price spike (this had never happened before). Fortunately, the software code that manages SlideShare’s cloud servers responded automatically by renting new machines at the “on-demand” rate, so we didn’t experience any actual downtime, only degraded service. But after this happened to us several times, we have changed the mix of machines that we use so that only half of them are from the spot market, and the rest are on-demand.

Spikes are a recent problem

Looking through the pricing history for various classes of machines, it’s clear that these spikes are new, and that they are happening across almost all instance types, at least for servers that are on the East Coast of the United Sates. For example, “small” servers on AWS both spiked as high as $100 an hour twice in November, when the on-demand price for those servers is $.085/hr. “m1.large” machines also spiked as high as $40 an hour. Almost every class of servers has hit spikes of more than 10 times their retail price in last few months. What is going on?

It’s hard to say why the spot market is suddenly showing more price spikes. A drop in supply (from Amazon requisitioning machines for its own purposes or for renting in the on-demand market) or a spike in demand (from the Christmas e-commerce rush) could be to blame. It’s important to remember that the AWS spot market is not a typical market, with many buyers and sellers doing business over a neutral exchange. One seller is servicing many buyers, and is also operating the exchange.

Amazon benefits from customer anxiety about getting access to spot servers: they sell on-demand instances for a higher price, and pre-paid reserved instances for better cash flow. So it’s unrealistic to expect Amazon to do anything to “fix” these price spikes. From Amazon’s perspective, they are a feature, not a bug.

How to deal with EC2 spot price spikes

For customers of the AWS spot market, there are some best practices to be learned from these recent price spikes:

• Never EVER bid more than you are willing to pay for a server on the spot market. This is the most important lesson. Don’t even bother doing “convenience bidding” of double or triple the on-demand price: when the price starts to spike it will easily go way beyond any rational price. Do you want to be the gal who explains to the CEO why the company is paying $100 an hour for servers?
• Don’t run all your infrastructure on spot market machines. In fact, don’t run more infrastructure than you are prepared to lose on spot machines. We use a thumb rule of 50 percent at SlideShare, since our system can easily survive 50 percent of our machines disappearing at one time (which is what will happen during a price spike).
• Write the code that manages your cloud infrastructure so that it responds intelligently to spot market price spikes. If you can’t get a spot machine at a reasonable price, your code should automatically request an on-demand server.
• Consider having some “reserved instances,” so that you are guaranteed the right to a minimum base level of machines. I’ve argued in the past that reserved instances don’t make sense for startups, but it’s clear that when supply dries up at Amazon it happens all at once, without warning. Your portfolio of servers on Amazon is almost like a financial porfolio. You want some diversification between risky high-reward elements (spot market) and more conservative elements (reserved instances).

These are early days for real-time pricing of cloud computing, and the spot market on Amazon is finally acting like a real market, with extreme price fluctuations. The “free ride” of getting reliable spot priced machines for less than the on-demand price is over. So if you want to play with cheap cloud servers, make sure you have the infrastructure in place to handle a price spike that could make all your servers vanish in the blink of an eye!

Jonathan Boutelle is co-founder and chief technology officer of Slideshare a web site for presentations that relies heavily on cloud computing. Previously, Jonathan was a principal at Uzanto, (a UI consulting firm) and worked as a software engineer at CommerceOne (a B2B enterprise software firm) and Advanced Visual Systems (a 3D graphics startup) You can find his presentations on cloud computing at slideshare.net/jboutelle, and his Twitter is @jboutelle. He also blogs at www.jonathanboutelle.com.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Migrating media applications to the private cloud: best practices for businesses
• Quality of the cloud: best practices for ISVs
• For UK education, private clouds may make economic sense

He told GigaOM that the actual job, which began about six months ago, involved a lot of slimming down of Skype’s native client and making it work in a browser, creating the back-end integration between Facebook and Skype and scaling up Skype’s network to handle the flood of traffic.

“We anticipate some really heavy usage, well in excess of what Skype sees today,” Rosenberg said.

Cut the client fat

The first big challenge was taking what is normally about a 20 megabyte native PC or Mac Skype client and stripping it down into a less-than-3-megabyte runtime, which works in conjunction with a JavaScript plug-in. Skype essentially ripped out much of the user interface and unneeded features of its Skype client to create a slim piece of software that could be quickly downloaded. The runtime provides some APIs, and the plug-in delivers those APIs through Javascript for access to Facebook.

Facebook’s team built the chat icons and video buttons, and it handles the download of the plug-in. Skype takes over during the actual call, logging into the plug-in and runtime and relying on its existing infrastructure to complete the call.

Where callers have no name

Rosenberg said 98 percent of Facebook video chats are identical to traditional Skype calls, save for some small rendering code for the video UI. In order to access Skype’s network, Skype actually has to create an anonymous Skype login for each Facebook video chat user, even if he has his own Skype login. A Facebook user’s identity must then be mapped to the Skype login to ensure that the call can go through.

That was something Skype spent time to get right, to make sure users were connecting to the people they intended to call and that their personal accounts were not able to be accessed by others, said Rosenberg. ”There were a lot of security things we needed to do to make sure it would just happen as users expected,” he said.

Many browsers = headaches

Getting Skype calls to work in a browser turned out to be more of a challenge, because Skype had to take into account how browsers operated. One of the biggest issues was dealing with the fact that users often can have several tabs open with Facebook. Skype had to be able to handle each instance of Facebook and ensure that the different tabbed versions were able to communicate correctly with the runtime.

“We knew this would be an issue, but it was more complicated than we thought,” Rosenberg said. “It turns out there is just a lot of concurrency use cases.”

Another issue was dealing with different browser and operating system combinations, making sure Facebook video chat would work on all configurations. That testing, along with performance tuning, took up much of the last month before Wednesday’s launch. Even now, Facebook video chat doesn’t work on Macs running the upcoming OS X Lion operating system, which was seeded last week to developers.

Scaling up behind the scenes

On the back end, Skype has bought more servers and bandwidth in its data centers to handle the extra traffic it expects. It still relies on its peer-to-peer network, the same video and audio codecs, for transport, but it needs more infrastructure to scale up reliablity for Facebook and handle the handshake between the two networks.

One key decision has been to rely completely on dedicated supernodes to facilitate Facebook video calls and to bring them all into Skype’s own data centers or into Amazon’s EC2 cloud. Supernodes select Skype clients that often sit on the public Internet, act as directories that help guide calls through the P2P network, and are usually used during recovery efforts after an outage. The fact that Skype is taking this step shows that it is serious about getting off to a good start on Facebook.

More features to come?

Skype’s integration opens up some interesting possibilites for future video chat services through Facebook. Rosenberg declined to say if Skype video chat will come to Facebook’s mobile apps or if group video chat will come to Facebook’s site. He said that Skype just rolled out its Android app and is still working on ramping that up.

But he said it’s technically possible, as is group chat for the browser. Bringing group chat to the desktop browser, however, would require an update to the runtime, which was slimmed down by pulling out features like group video calls and instant messaging. It might take a little time, but I don’t doubt that we’ll see both features come in the future. Skype CEO Tony Bates said on Wednesday that the integration was a chance to introduce Facebook users to Skype’s paid products, things like group video chat.

You can read more about the Skype-Facebook integration on Skype’s blog.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The Future of Workplaces
• A 2011 Connected Consumer Forecast
• A 2011 NewNet Forecast

Twitter users can simply go to http://theirtwitterusername.fyels.com to claim their personal drive, then authenticate their accounts through Twitter’s OAuth implementation. Fyles also includes embedded audio and video players for various media file types, making it a great solution to quickly share video footage via Twitter. Check out my own Fyels drive here.

Fyles is being developed by Omnissient, a company based in Brisbane, Australia, and Omnissient Director Dominic Holland told me via email that Fyles was originally meant to be a tool to help with the development of other projects. “Being developers, we are constantly moving files between computers and servers, and we wanted a fast and free way to do that without time, or monetary restraints,” he said.

However, Fyles took off right away, and is now well on its way to attracting 80,000 registered users within one month of its launch, said Holland. His company took this as a sign to concentrate on the service and sideline all other projects.

Fyles utilizes HTML5 for its user interface, and the back-end is also state of the art: “The platform utilizes Redis/NOSQL for the database engine, hence the speed”, explained Holland. “It runs across a number of large EC2 instances, and storage is hosted with S3, and we utilize the Cloudfront CDN for all downloads.”

Holland told me his company wants to offer the service ad-free for the time being. “We do have monetization strategies in mind, but they are absolutely not being a key focus for us at this stage,” he said, adding that the main goal for now is to grow the site’s user base.

To be honest, I’m a little skeptical how sustainable this will be. I’ve seen plenty of startups offer free file hosting plans, only to quickly scale down its offering in face of exorbitant bandwidth and storage bills. S4ve.as and Podmailing are only two that come to mind. That being said, Fyles does look like a great service, so let’s hope it stays around for at least a little while.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• In Q4, Data Centers, Not the Cloud, Were the Big Story
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Report: NoSQL Databases – Providing Extreme Scale and Flexibility

In December, Microsoft reorganized by forming a Server and Cloud division, following a slow and steady rollout of Microsoft Azure throughout 2009. The updates included all the usual tactics of getting developers, service providers, and early customers on the bandwagon. In November, Microsoft officials held court at the company’s Professional Developers Conference, seeking to engage the development community. It became clear there that while Azure does not equal Windows, common development frameworks like .NET deliver a more seamless bridge between on-premise and cloud deployments than existed previously.

Microsoft’s cloud emergence is visible elsewhere, too. Amazon recently announced support for Microsoft Server 2008 on EC2, and the company has been offering at least some Windows support since late 2008. In November,  Amazon announced support for a software development kit for .NET developers that “provides a set of developer-friendly APIs that hide much of the low-level plumbing associated with programming for the AWS cloud, including authentication, retries, and error handing.” Rackspace has also announced Windows support is coming, with a beta release slated for early 2010.

Other factors are behind Microsoft’s cloud focus as well. One is the Microsoft Service Providers License Agreement (SPLA), which was updated in 2009 to make it easier for service providers to offer Windows options more cost-effectively. While there’s some debate in the service provider community about how the SPLA actually works and whom it benefits, Microsoft has provided more options than in the past, giving cloud service providers better flexibility.

It’s also worth considering the sheer volume of tools and resources that Microsoft is wrapping around Azure. As Stacey outlined in “Microsoft Azure Walks a Thin Blue Line,” there are plenty of tricks up Microsoft’s sleeve for supporting enterprises and service providers, including AppFabric, which “helps developers connect applications and services in the cloud or on-premise.” Microsoft is also going all out to provide interoperability with programming frameworks and software such as Ruby on Rails and MySQL.

Finally, there’s the issue of what enterprises really want from the cloud. While there’s been plenty of discussion about virtualization at both the enterprise and service provider level, Windows commands a far greater footprint than any hypervisor out there. It appears unlikely that many CIOs favor Windows administration as a long-term core competence. Perhaps those CIOs can offload some of their more basic Windows applications directly to the cloud first. That’s just one more way that Microsoft’s business model and dependence on success in the cloud could shift dramatically next year.

Image courtesy of TechFlash Todd on Flickr.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• In Q4, Data Centers, Not the Cloud, Were the Big Story
• Architectural Shift: Clouds and GPUs As the Future of HPC
• VMware’s Cloudy Ambitions: Can It Repeat Hypervisor Success?

You might think the deal is just cloud roll-up: Reasonably Smart was a very small startup. David Young, Joyent’s CEO, said the company–whose backers include PayPal co-founder Peter Thiel– is in “a strong financial position that supports making strategic acquisitions.” Dig a bit deeper, however, and the deal is more than just a roll-up. Joyent gets an open platform with which to attract developers while preparing the company for the looming threat of Google and Microsoft.

Joyent, along with other infrastructure-centric clouds like Amazon’s EC2 and Rackspace’s Mosso, let subscribers see their machines. Because they virtualize at the hardware level, these clouds support a wide range of development languages. Users aren’t locked in: They can take their applications out of the cloud and run them themselves. But this model also means customers have to worry about operating their virtual infrastructure, undermining the promised scalability of cloud computing.

By contrast, service-centric cloud models like Google’s App Engine, 10Gen and Microsoft’s Azure hide the infrastructure from developers. A subscriber doesn’t worry about scaling. Instead, they fret over lock-in — the inability to leave their cloud provider when things go wrong because they’re dependent on its proprietary features.

With Reasonably Smart, Joyent can strike a balance between infrastructure and service. Developers write applications in JavaScript, using extensions for things like I/O and storage. These applications can run on a developer’s desktop, in a private data center, or in a cloud. Of course, Joyent’s betting its operational expertise will convince people to run it in their cloud. It’s a service model, but one that subscribers can leave if they want to.

Joyent CTO Jason Hoffman remarked last year at our Structure conference that he wanted an open cloud model that could scale indefinitely, independent of infrastructure concerns. “We intend to keep this new Joyent offering completely open-source,” he said of today’s acquisition.

The move toward service-based clouds is part of a trend that will shape cloud computing in 2009. This year, Google and Microsoft will roll out production-grade clouds that have features like search, mapping, licensing, social graph and authentication baked right in.

To compete, infrastructure clouds need to round out their open offerings with built-in services while trying to retain the openness of their infrastructure heritage. Amazon, which has plenty of services, from SimpleDB to S3, continues to extend its offering with value-added services like CDNs, billing and management consoles — even at the expense of its ecosystem of existing vendors that have built similar tools atop EC2.

That makes Joyent’s acquisition look, well, reasonably smart.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Microsoft Azure: What It Is, What It Costs and Who Should Care
• In Q4, Data Centers, Not the Cloud, Were the Big Story
• Big Data 2011 Preview

F|R: Let’s say you’re about to start, or fund, a new business. Considering the changing landscape for computing architecture, what emerging or ignored problem in cloud computing would you target? What business or service would you launch to try to address it?

Arnie Berman, Chief Technology Strategist, Cowen and Co.:

Since Structure 08, I’ve been mulling over the following conundrum: How to give those that sign up for cloud computing services a sense of just how big (or small) their bills will be. In the wireless handset world, the advent of “bucket pricing” greatly helped the cause of mobile adoption. With “per-minute” pricing plans, charges were unpredictable – and occasionally very painful for active callers. “Bucket pricing” removed the fear of getting stuck with a ridiculously large bill.

In enterprise software, the advent of monthly subscription pricing from SaaS vendors likewise disrupted the license and maintenance pricing model of conventional software. But now companies like Amazon are threatening to disrupt the monthly pricing model with a “by-the-drink” model, where users only pay for the computing or storage capacity they use. While the economics of a pay-as-you-go approach are extremely compelling for most users, the approach actually reintroduces an element of uncertainty, because it’s very hard to predict what your computing consumption, and therefore your spending, is going to be. In wireless handset terms, it’s like going backwards from the safe harbor of “bucket pricing” to risky “per-minute” plans again.

Here’s a graphic:

Users like saving money. But they also like predictability. This suggests that what the world needs now is a monitoring and provisioning tool: a service (or a ware) that allows users to forecast their likely usage of computing infrastructure resources that they’ll purchase “by the drink.” Such a tool would also help predict their likely savings, as compared to the traditional on-premises software or SaaS models. Think of this tool as a tech innovation to help measure, and manage, the business model innovation that is cloud computing.

Luke Lonergan, co-founder and CTO of Greenplum, a maker of database management software:

The hype of cloud computing is huge with promise, and there are a lot of companies trying to explore what they can do with it. It is creating a market for services around putting applications into Amazon EC2 (leased computing infrastructure) and S3 (online storage) quickly and easily. If I were a VC in the market for a cloud company right now I’d look for something that makes the path from “application” to “cloud application” much quicker — basically I’d sell shovels and alcohol to the gold miners.

As far as the characteristics required: It would be off-the-shelf service for quick experimentation, something like “put your code or job stream here, dial in the number of units, push button to run, back comes a monitor that watches over execution, when done you get stats about efficiency, etc.” I think the current Amazon interface is more about the rental of assets and less about the execution of the experiments themselves.

Ken Elefant, General Partner, Opus Capital:

The most successful startups will identify enterprise computing needs that are not core to their customer’s business, but are still required to support it. This means targeting companies where a business buyer is eager to bypass IT. And founders should be going after cyclical businesses with predictable spikes. These are the kinds of potential customers that can benefit the most from overflow IT provisioning offered by a cloud computing startup. For example, search companies would want their large amounts of data to be in the cloud (pictures, videos, etc.) but only if they have low latency, low cost and high availability.

But don’t be too narrow in your pursuit of suitable verticals. Small companies still need to prove they have broad market applicability — and, specifically — enterprise customers who are willing to pay. The verticals need to be large enough to warrant a venture investment. An example might be financial services, where the provisioned service or function that a cloud computing startup might offer would be risk calculations, where lots of data, low latency and high availability is essential. Financial services is also a good example of a vertical of willing buyers. I believe technology will continue to trend in favor of talented, visionary entrepreneurs, especially those who can move fast.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A 2011 Infrastructure Forecast
• In Q4, Data Centers, Not the Cloud, Were the Big Story
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4

“Most large-scale systems, in order to move up the ladder and serve more customers, require more and more resources,” said Bert Armijo, 3Tera’s VP of product and marketing. “If you manage them as individual virtual machines, the problem is that the human load — the ability to actually remember what’s running where and to manipulate it all — becomes overwhelming. At some point, somebody makes a very small mistake that results in a very large outage.”

3Tera’s Applogic makes software that runs on a grid of hardware: A flat array of commodity servers, Gigabit Ethernet and direct-attached storage. The software turns this into a resource pool that can be provisioned to users. A graphical front-end, called an infrastructure editor, lets administrators drag and drop data center components like firewalls and load balancers.

3Tera sells to enterprises that want to run their own grids, as well as to roughly 20 managed service providers that want to offer on-demand data centers to their customers. Pricing starts at $500 a month for the smallest virtual data center, which is a single 8-core node, 8GB of RAM, 750 GB of storage and 2TB of transfer, though the company recommends a minimum of a three-server configuration. Within that, the manager can slice up the virtual resources into 100 individual servers in the VDC. In recent months, the firm is increasing utilization at roughly 15 percent per month.

3Tera was bootstrapped on its founders’ pocketbooks, but in April it took in a first financing of $3.7 million; the largest investor in the round was Japanese distribution partner Net One Systems. At the same time, the firm announced plans to open up its software to work with third-party components by unbundling its software into execution, catalog and control components. “We’ve proven out the ability to move applications between data centers and to abstract infrastructure definition from physical hardware,” said Armijo. “Now, somebody could write a connector between our control interface and the [Amazon] EC2 execution engine.”

3Tera has strong parallels to firms such as Elastra. While they look like competitors, Armijo says he thinks VDC companies need to cooperate. “It’s in our interest to show interoperability and the ability to run applications that span both systems. This is what will inspire users to put their applications into the cloud.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A 2011 Infrastructure Forecast
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4
• Defining Internal Cloud Options: From Appistry to VMware

• Sun’s OpenSolaris OS will be available on the Amazon Elastic Compute Cloud (Amazon EC2) customers for free. It is in beta for now.
• Sun will provide premium technical support for MySQL database running on Linux and Amazon EC2.

These developments are meant to address the needs and complaints of the developer community. OpenSolaris, which comes with tools such as ZFS and Dynamic Tracing (D-Trace), will be offered for free, in contrast to some Linux offerings that cost money. For instance, if you sign up for EC2 and pick RedHat, it costs $19. ZFS allows instant rollback and continual check-summing capabilities, something developers have found lacking in the EC2 platform. This OpenSolaris on Amazon EC2 beta is currently available by invitation only. Some software vendors, including GigaSpaces, Rightscale, Thoughtworks and Zmanda, are already offering their solutions via Amazon Machine.

From OStatic: As Sun Microsystems’ JavaOne conference kicks off this week, the company has announced its free new OpenSolaris open source operating system. It’s available for download . The big question with OpenSolaris is how it may compete against Linux rivals, especially since it is a fully supported operating system. OStatic, our open source blog, has the details.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• In Q4, Data Centers, Not the Cloud, Were the Big Story
• A 2011 Infrastructure Forecast
• Big Data, ARM and Legal Troubles Transformed Infrastructure in Q4