The theory behind tools like these is basically this: sure, public cloud computing is billed as cheap, but too often it turns into a wasteland of dormant instances and other fallow resources. So as inexpensive as it can be, it’s not necessarily efficient or as cheap as it could be. Companies like Newvem, Cloudability, Cloudyn, CloudVertical et al say they can help you optimize all that and save more.

Newvem for Windows Azure covers many of the same core usage and cost metrics as the AWS version. A “heat map” helps users visualize their workloads as they move from on-premise implementations to the cloud, according to Newvem VP of marketing Cameron Peron. The free beta is available now to all Azure users. Newvem’s AWS version started out free as well, and a base level of capabilities remain free, but as of late last year, the company started charging for higher-level services.

Newvem said it sees Azure — which launched its AWS-like Infrastructure-as-a-Service  capabilities last month — gaining traction.

“The size of the Azure installed base is probably one of [Microsoft's] best-kept secrets,” Peron noted. Well, not that secret since Microsoft recently said Azure is a $1 billion-a-year business – a claim that some find difficult to swallow. Newvem would not comment when asked if Microsoft helped fund its Azure tool, but given that Microsoft wants to build the Azure ecosystem and compete better with AWS (as well as the spanking new Google Compute Engine), I’d say it’s a safe bet.

It’s also true that companies like Newvem, which built services around AWS, have been perplexed to see AWS adding richer and deeper monitoring and management services like Trusted Advisor. Given that, it makes sense that these companies offer multi-cloud capabilities.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud computing infrastructure: 2012 and beyond
• Infrastructure Q1: Cloud and big data woo enterprises
• Forecasting the future cloud computing market

For example, the company is looking to hire a  software development manager for its “Kumo team” who will manage:

” .. software development engineers who are developing algorithms and building systems to automatically solve a variety of Information Retrieval and Data Mining problems related to the AWS Trusted Advisor — one of the company’s biggest assets.”

The Kumo team is also looking for a software development engineer and a senior product manager.

For enterprise support, more is more

That AWS should seek to add richer functionality to Trusted Advisor should come as zero surprise. The boilerplate describing the service says AWS will continually add to the number of checks it performs.

Reached for comment, a spokeswoman said AWS is investing heavily in support “not just with Trusted Advisor but also with the Service Health Dashboard, Forums, Social Media Monitoring, Service Health Checks Support APIs” and other tools that allow its engineers and associates to deliver a good user experience.

As for “Kumo,” it means cloud in Japanese and is the name of the AWS Support software development team.

But what about third-party partners?

It’s clear that as AWS wants to make itself as enterprise-friendly as possible, providing services like these is a no brainer. But, for a half dozen or so small companies that were founded to provide cloud implementation guidance, particularly on AWS, this has to be a concern. Companies including Cloudyn, Cloudability, CloudVertical, and Newvem all harvest — with customer permission — usage data from Amazon’s APIs and use that to make their recommendations. Some of them monitor multiple clouds. Others are for AWS only, a strategy they want to reconsider going forward.

Earlier this month, AWS made a support API available that, among other things, gives customers API access to their Trusted Advisor recommendations. That was seen as a huge deal by some in the AWS partner community.

Some of these partners flipped out when Amazon made Trusted Advisor available for free for a short period. They really don’t like that it’s now making all that key data programmatically available.

Said the CEO of one of these companies: “This is another step, in a roundabout way, of AWS giving their customers a Cloud Economics toolset. I’m not sure they haven’t just head-on launched a Cloud Economics tool — users need quite a lot of expertise to hack together the various data sets you can get exported to S3 – and now this. I suppose that’s the short-term opportunity for companies like ourselves.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Understanding and managing the cost of the cloud
• Infrastructure Q4: Big data gets bigger and SaaS startups shine

According to a survey by Cloudyn and The Big Data Group of 450 Cloudyn customers — who all use AWS — here are the main takeaways:

• Amazon’s EC2 constitutes nearly two-thirds (62 percent) of total AWS spending.
• More than half of those EC2 users now deploy Reserved Instances as part of their deployment.
• On-demand pricing remains the number one choice for most users — it sucks up 71 percent of all their EC2 spending.
• The S3 Simple Storage Service is still the most popular storage option, although Glacier, the cheaper archival storage choice, is gaining momentum.
• The largest constituency among the Cloudyn/AWS users are those who spend less than $50,000 a year on AWS, but they account for just 4 percent of total AWS spending.
• Just 4 percent of the customers spend more than $1 million a year on AWS, accounting for 52 percent of total spending.
• Customers who spend less than $50K per year make up the largest group of AWS users, yet account for only 4 percent of total spend.

Cloudyn is one of a half dozen or so startups that have made tracking, monitoring and managing AWS infrastructure their business. Competitors include Cloudability, Newvem and RightScale. And all of them are eager to prove that they can save their customers the most money by guiding their AWS deployment choices. Of course AWS itself is not standing still, building more granular monitoring and management options — including Trusted Advisor –  over time.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud computing 2013: how to navigate without a map
• Cloud and data third-quarter 2012

Its AWS Cost Analytics will help users make sense of a deluge of usage data that most now have to put into unwieldy spreadsheets to manage, the company said.. “With this tool, you will no longer have to pump giant files — tens of megabytes in size — and pares them down so you don’t have to pump them into Excel,” said Cloudability CEO Mat Ellis.

Cloudability CEO Mat Ellis.

Ellis says this tool differs from what Cloudability competitors now offer because it provides not just “pre-canned” reports but will let the user build “17 million different combinations of reports as needed” all based on this burgeoning flow of cost information provided by Amazon.

To be sure Cloudability faces a half dozen or more competitors – Newvem, Cloudyn, Cloud Vertical, and CloudChekr – in this AWS cost assessment and management space – all offering their own take on what goes on in your Amazon cloud. What Cloudability says its doing differently is providing more user-defined customization and the ability to handle this new influx of hourly data.

Ron Fuller, web manager for Mentor Graphics, a large electronic design automation vendor, is sold. Other tools, provide simpler reports and alerts but by the time the alert occurs, you are probably already over budget, he said.

“If I have a $15,000 a month to spend and hit that limit half-way through the month, I can’t just shut down. Cloudability gives us daily reports with our incremental usage spend and detailed cost analysis across multiple accounts and projects,” Fuller told me. That can flag problems before they get out of hand.

Like many AWS users, Fuller loves what he’s able to do with all those AWS services, but the complexity of tracking their usage can be overwhelming. Cloudability’s tool helps him figure out where to use discounted reserved instances vs. other, pricier instance types for example. “If I’m overspending in areas of I/O peformance, I can see that right away and maybe rethink my test model.”

Cloudability and its rivals have to contend with each other but increasingly with the Amazon mothership itself which keeps adding more management and assessemnt tools of its own. But, as Forrester Research analyst Dave Bartoletti said few months ago: “Amazon’s tools will get better and better but Amazon has no desire to get you to use less of its services. It’s like in storage — You’d think EMC would be the best vendor of storage management but historically they haven’t been.”

Keep in mind: All these  third-party vendors rely on Amazon-supplied data to work their magic, and given the rather bracing competition so far, I would expect the others to add similar reporting capabilities pretty quick.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data third-quarter 2012
• Understanding and managing the cost of the cloud

Remember last July when our own Om Malik and Stacey Higginbotham reported on VMware’s plan to take on Amazon with a cloud of its own?  Remember VMware denying it?  Well, last Wednesday it all became official with the announcement of VMware Hybrid Cloud services — in which VMware will host its own public cloud that will be sold by existing VMware partners.

Gartner’s Chris Wolf has an interesting take on his blog here.

Now, there is some wiggle room for VMware here. Our original story showed the big cloud coming out of the then-unannounced spin-off of VMware and EMC That spinoff — the Pivotal Initiative — did happen, but it appears that it’s VMware, not Pivotal, that’s running the big cloud. Paul Maritz the former CEO of VMware and now the head of Pivotal, will be speaking this week at GigaOM’s Structure: Data in New York.

Amazon beefs up Virtual Private Cloud

It was an even more busy week than normal for Amazon itself which unveiled new Virtual Private Cloud capabilities for AWS users. Launched 4 years ago Amazon VPC lets users create “a virtual network of logically isolated EC2 instances and an optional VPN connection to your own data center,” according to the AWS blog. what’s new now is that soon, when a new customer launches EC2 instances, they will do so in the ”EC2-VPC” platform.

“You don’t need to create a VPC beforehand – simply launch EC2 instances or provision Elastic Load Balancers, RDS databases, or ElastiCache clusters like you would in EC2-Classic and we’ll create a VPC for you at no extra charge.  We’ll launch your resources into that VPC and assign each EC2 instance a public IP address.”

At that point you can assign multiple IP addresses to a single instance, change security group membership on the fly, and add egress filters to those groups.

One of Amazon’s weak points when it comes to the enterprise accounts it wants to attract is the unwillingness of IT pros — especially in heavily regulated businesses — to put mission critical workloads on shared public cloud infrastructure. This new VPC capability might make them more amenable to using AWS at a time when more and more “private” cloud options — from competitors Rackspace, HP, and others are coming on line. GCN has more on VPC.

Also new his week: AWS tripled the amount of storage that can be associated with each RDB (Relational Database) instance.

“You can now create DB instances (MySQL or Oracle) with up to 3 TB of storage (the previous limit was 1 TB) and 30,000 IOPS (previously, 10,000). SQL Server DB Instances can be created with up to 1TB of storage and 10,000 IOPS.”

And, Amazon also launched a version of its mobile AWS console for iPhone so It types can see what’s going on with their Amazon cloud when they’re not at their desks. An Android version of the console debuted in January.

This public cloud war will only get more interesting if, as has been reported here and elsewhere, IBM is in the market for SoftLayer and/or Rackspace.  If IBM ends up making either of those rather substantial purchases — Reuters put a $2 billion price tag on SoftLayer and Rackspace would be much more — it would only show how critical IBM sees this market that it would spend that kind of dough after already investing billions in its own cloud infrastructure. Of course, no one — from IBM, from SoftLayer, from Rackspace or from EMC (which was also reportedly in the hunt for SoftLayer), is commenting.

Working with your Frenemy: Newvem debuts AWS heat map

Newvem is the energizer bunny of AWS watchers. Amazon keeps “totally copying” Newvem’s capabilities (in the words of Newvem CEO Zev Laderman) and Newvem just keeps on adding more capabilities.

Newvem, the Israeli company obsessed with showing you more about your Amazon Web Services use than you know, was at it again this week, unveiling its Cloud Utilization Heat map that shows AWS users their utilization over multiple regions by time, machine type and availability zone.

Newvem CEO Zev Laderman (left) and AWS CTO Werner Vogels.

The sales pitch?A substantive portion (15 percent) of the clouds Newvem analyzes now are underutilized. That means “$30 million of $200 million EC2 spending tracked is wasted,” says Newvem spokesman Cameron Peron. Presumably if you know what resources are being wasted you can make moves to streamline operations and run things optimally. Sell off reserved instances you’re not using, for example.

Here’s my guess: Newvem has focused 100 percent on AWS to this point. But as Amazon keeps adding more of its own dashboard and monitoring alerts, I would bet Newvem, like other monitoring companies, will soon turn its eye to other clouds as well. After all, multi-cloud deployments are one tactic large companies are looking at to avoid locking into a single provider.

That Amazon competes with its own partners is certainly not new. It’s been going on for years as we’ve reported. The company, like Microsoft or IBM before it, treads a fine line between relying on smaller, nimbler partners to come up with innovative new services, and then adding more value-added services itself. Amazon launched a partner program last year to help navigate this terrain.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Takeaways from the second quarter in cloud and data
• Microsoft Azure: What It Is, What It Costs and Who Should Care
• Metered IT: the path to utility computing

News of an updated version of Trusted Advisor — complete with new features and its free trial (for the month of March) — was unveiled on the AWS blog early Monday morning. Before now, Trusted Advisor was available to customers who signed up for enterprise or business class AWS support.

According to the blog, Trusted Advisor looks over a customer’s AWS environment and makes suggestions on how to save money, boost performance and shutter security gaps:

“Because the AWS Trusted Advisor draws upon the aggregated operational history of hundreds of thousands of AWS customers, you can be confident that the recommendations that it makes can help you to save money, bolster your security profile, improve the fault tolerance of your application, and increase overall performance. This is a unique and powerful benefit that is only possible with cloud-based, API-enabled infrastructure.”

Meanwhile, companies like Newvem, Cloudyn, Cloud Vertical and Cloudability have to be more than a little worried about this new tool, although they’d be the first to tell you that their own respective offerings watch and measure AWS better than Amazon itself does.

A Newvem spokesman characterized the freebie as big news for AWS users and “a great value as a broken-to-fix support play as in something is wrong with my security, I’ll use Trusted Advisor to fix it.” But, he added, Newvem provides more insights on how to improve a user’s AWS resource usage and to evaluate costs, risks and assets. Newvem started charging for its service late last year.

Amazon’s news is a no-brainer for a company that knows it needs to provide more enterprise-class support and monitoring options to placate enterprises used to having such tools, as GigaOM reported last summer. But it also illustrates the issue that, to grow, Amazon is encroaching more and more on spaces pioneered by small members of its ecosystem. Being an AWS technology partner is a risky proposition that is not for the faint-of-heart or the slow-of-foot.

What about the little guys?

Usually, when small companies characterize a huge company’s incursion into their territory as a validation of their strategy, it’s time to pat them on the head and offer condolences. In this case, however, there is some truth that a smaller, more nimble third party (aka Newvem, Cloudyn, et al) can offer more value.

As Forrester Research analyst Dave Bartoletti told me last month with regard to some Cloudyn news:

“Amazon’s tools will get better and better but Amazon has no desire to get you to use less of its services. It’s like in storage — You’d think EMC would be the best vendor of storage management but historically they haven’t been.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Understanding and managing the cost of the cloud
• Cloud and data third-quarter 2012

Newvem CEO Zev Laderman (left) with Amazon CTO Werner Vogels

Large enterprises will negotiate their own deals, but for smaller accounts the company will offer free services until the customer goes over 50,000 AWS resource hours per month.  Then it charges a cent or two per additional resource hour depending on usage. According to Newvem’s price list, there are additional fixed-rate charges for more advanced analytics of S3 storage or EC2 reserved instance use.

One service the company is particularly proud of is tracking AWS usage by department or business function within a large organization. “Amazon itself is very horizontal — it’s just infrastructure — they’ll tell you how many instances and how much storage you use all in one bill. What we do is let you slice and dice Amazon resources against your divisions — no one else can do that,” said Newvem president Zev Laderman. (Update: For the record, Cloudability disputes this, saying it offers similar services.) “If you have R&D and engineering and manufacturing folks and want to tie all AWS usage to each unit to make sure they’re consuming it properly, if there are security issues or vulnerabilities with how ports are setup, if you want to charge-back from a central unit, you can do all that.” 

It’s getting crowded in here

There are a dozen or so companies in this AWS stalking market and they all claim to do what the others cannot. Cloudability, which now says it monitors $250 million in cloud spending,  charges customers a percentage of their overall cloud spend after a free 30-day period. Dublin-based CloudVertical offers similar tiering. Cloudyn offers basic services for free, then tiered pricing for more advanced services above that. There will be more third-party analytics and reporting services coming down the pike.

One newbie is New York-based CloudAware, an AWS consultancy that has done monitoring and analytics work for AOL, Sony and Reed Elsevier (see disclosure) and is now looking to sell a service based on that work. CloudAware will provide a service that alerts management, via a news feed-like feature, to any changes in configurations to AWS resources, said CloudAware CEO Mikhail Malamud. “If someone launches a firewall change and that change is on a non-standard port that will kick off an approval request,” Malamud said.

Mission: Making AWS less inscrutable

These Amazon partners all fulfill a need by enterprise customers to get a better grasp on what many see as a black box.  For one thing, Amazon services are billed as cheap and easy, but cheapness is all relative. At least one Amazon expert experienced unexpectedly high charges  moving S3 data to Amazon’s cheaper Glacier data archiving service for example. So there is a real need for services that monitor and alert customers about their AWS usage.

Nuances aside, these companies all compete with each other. But probably more worrisome longer term for  the startups in this space is that Amazon itself keeps adding deeper and broader billing and analytics capabilities. Offerings like its Trusted Advisor service, now in beta and more granular billing, are eating away at territory these small companies have staked out.

The next year or so will, for sure, be an exciting ride for partners in the Amazon ecosystem.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in Giga Omni Media, the company that publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data third-quarter 2012
• Understanding and managing the cost of the cloud

A letter went out today to AWS customers about this new capability. Text below:

Dear AWS customer,

We are excited to announce Detailed Billing Reports, a new hourly grain view of your AWS usage and charges. This detailed report enables you to better understand your AWS Bill by providing hourly usage and cost data by product and Availability Zone. In addition, consolidated billing customers can now view unblended rates and cost. This report is particularly useful for analyzing your usage of Amazon EC2 On-demand and Reserved Instances.

To get started, visit the Billing Preferences page to sign up for Detailed Billing Reports. You will also need to sign-up for Programmatic Access, as reports are only published to your Amazon S3 bucket. AWS generates new reports based on your estimated bill multiple times per day and a final report for month end.

To learn more about Detailed Billing Reports, please visit About AWS Account Billing.

Sincerely,
The Amazon Web Services Team

Amazon posted a blog about this new capability as well. This, along with Amazon’s Trusted Advisor service, offers customers the kind of visibility that a raft of smaller Amazon ecosystem partners such as Cloudability, Cloudyn and Newvem are trying to provide. Stay tuned for updates.

Update: Newvem CEO Zev Laderman says Amazon’s new transparency compliments his company’s services.  ”Since we are analyzing usage — risks, costs, and assets — better insight into costs and billing from AWS offers a combined approach for end-users to improve their usage,” he said via email.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• The role of converged infrastructure in the data center
• Metered IT: the path to utility computing

Cloudyn CEO Sharon Wagner.

The latest partnership is between Cloudyn, which captures and analyzes info about user’s AWS utilization — cost, performance and utilization of EC2 instances etc.  – and Scalr which helps users’ deploy and manage their cloud over time. Under a new pact, customers can see Cloudyn’s breakdown of their AWS usage on their Scalr dashboard  and get recommendations on a course of action. Once they choose an option, Scalr will automate it in one click, Sharon Wagner, CEO of Cloudyn told me via email.

Scalr itself supports Amazon, Eucalyptus, Cloudstack, Nimbula and Rackspaceclouds but the initial joint offering is for AWS only.

Newvem, another company that watches AWS usage for customers announced a slate of partnerships at AWS: Reinvent two weeks ago. Its partners include Cedexis, Cloudant, Datapipe, Garantia Data and others.

Other startups in this cloud monitoring field also include Cloud Vertical and on Wednesday, Apptio joined the fray with a free Apptio Cloud Express for tracking AWS, Windows Azure and Rackspace workloads and making optimization recommendations.

All of these companies hope to carve off a piece of a growing market. Gartner estimates that enterprise spending on public cloud services will grow to $207 billion in 2016 from $109 billion this year. But cloud watchers will have to duke it out not only with each other but with the cloud providers themselves as they add more analytics to their own services. Amazon, for example, is beta testing a Trusted Advisor service which provides many of the same metrics as these third parties.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• In Q3, Big Data Meant Big Dollars
• Understanding and managing the cost of the cloud

As of Thursday, Newvem is making its analytics broadly available, just a few days after Cloudability did the same.  Newvem said it’s gained valuable insights from the 8 months its free service has been in trial.  First and foremost “we’ve learned that cloud users are sick and tired of hearing about saving costs … we’re helping cloud users [instead] focus on their business and how the cloud impacts profit,” said Cameron Peron,  Newvem VP of marketing.

The Israeli company measures all Amazon resources used by a given company in real time and uses that data to recommend better deployment options.  A dashboard exposes key performance indicators (KPIs) for C-level executives, IT managers and developers, Peron said.

This market for analytics and dashboards into what some call Amazon’s “black box” is getting pretty crowded.  Not only are there a raft of feisty startups, but established players like Rightscale are starting to get into the act with their own tools. But the biggest factor on the horizon may be Amazon itself  which is starting to make its own costs more trackable.

If you’re confused about which of these tools to use, all of these companies, including Newvem, will be at Amazon’s ReInvent show later this month in Las Vegas. You can check them out for yourself and also hear what Amazon CEO Jeff Bezos and CTO Werner Vogels have to say. At Structure Europe last month, Vogels promised all sorts of “unbelievable” new services to come from the company some of these are bound to be announced at the show.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Understanding and managing the cost of the cloud
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses