For example, the company is looking to hire a  software development manager for its “Kumo team” who will manage:

” .. software development engineers who are developing algorithms and building systems to automatically solve a variety of Information Retrieval and Data Mining problems related to the AWS Trusted Advisor — one of the company’s biggest assets.”

The Kumo team is also looking for a software development engineer and a senior product manager.

For enterprise support, more is more

That AWS should seek to add richer functionality to Trusted Advisor should come as zero surprise. The boilerplate describing the service says AWS will continually add to the number of checks it performs.

Reached for comment, a spokeswoman said AWS is investing heavily in support “not just with Trusted Advisor but also with the Service Health Dashboard, Forums, Social Media Monitoring, Service Health Checks Support APIs” and other tools that allow its engineers and associates to deliver a good user experience.

As for “Kumo,” it means cloud in Japanese and is the name of the AWS Support software development team.

But what about third-party partners?

It’s clear that as AWS wants to make itself as enterprise-friendly as possible, providing services like these is a no brainer. But, for a half dozen or so small companies that were founded to provide cloud implementation guidance, particularly on AWS, this has to be a concern. Companies including Cloudyn, Cloudability, CloudVertical, and Newvem all harvest — with customer permission — usage data from Amazon’s APIs and use that to make their recommendations. Some of them monitor multiple clouds. Others are for AWS only, a strategy they want to reconsider going forward.

Earlier this month, AWS made a support API available that, among other things, gives customers API access to their Trusted Advisor recommendations. That was seen as a huge deal by some in the AWS partner community.

Some of these partners flipped out when Amazon made Trusted Advisor available for free for a short period. They really don’t like that it’s now making all that key data programmatically available.

Said the CEO of one of these companies: “This is another step, in a roundabout way, of AWS giving their customers a Cloud Economics toolset. I’m not sure they haven’t just head-on launched a Cloud Economics tool — users need quite a lot of expertise to hack together the various data sets you can get exported to S3 – and now this. I suppose that’s the short-term opportunity for companies like ourselves.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Understanding and managing the cost of the cloud
• Infrastructure Q4: Big data gets bigger and SaaS startups shine

According to a survey by Cloudyn and The Big Data Group of 450 Cloudyn customers — who all use AWS — here are the main takeaways:

• Amazon’s EC2 constitutes nearly two-thirds (62 percent) of total AWS spending.
• More than half of those EC2 users now deploy Reserved Instances as part of their deployment.
• On-demand pricing remains the number one choice for most users — it sucks up 71 percent of all their EC2 spending.
• The S3 Simple Storage Service is still the most popular storage option, although Glacier, the cheaper archival storage choice, is gaining momentum.
• The largest constituency among the Cloudyn/AWS users are those who spend less than $50,000 a year on AWS, but they account for just 4 percent of total AWS spending.
• Just 4 percent of the customers spend more than $1 million a year on AWS, accounting for 52 percent of total spending.
• Customers who spend less than $50K per year make up the largest group of AWS users, yet account for only 4 percent of total spend.

Cloudyn is one of a half dozen or so startups that have made tracking, monitoring and managing AWS infrastructure their business. Competitors include Cloudability, Newvem and RightScale. And all of them are eager to prove that they can save their customers the most money by guiding their AWS deployment choices. Of course AWS itself is not standing still, building more granular monitoring and management options — including Trusted Advisor –  over time.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud computing 2013: how to navigate without a map
• Cloud and data third-quarter 2012

Its AWS Cost Analytics will help users make sense of a deluge of usage data that most now have to put into unwieldy spreadsheets to manage, the company said.. “With this tool, you will no longer have to pump giant files — tens of megabytes in size — and pares them down so you don’t have to pump them into Excel,” said Cloudability CEO Mat Ellis.

Cloudability CEO Mat Ellis.

Ellis says this tool differs from what Cloudability competitors now offer because it provides not just “pre-canned” reports but will let the user build “17 million different combinations of reports as needed” all based on this burgeoning flow of cost information provided by Amazon.

To be sure Cloudability faces a half dozen or more competitors – Newvem, Cloudyn, Cloud Vertical, and CloudChekr – in this AWS cost assessment and management space – all offering their own take on what goes on in your Amazon cloud. What Cloudability says its doing differently is providing more user-defined customization and the ability to handle this new influx of hourly data.

Ron Fuller, web manager for Mentor Graphics, a large electronic design automation vendor, is sold. Other tools, provide simpler reports and alerts but by the time the alert occurs, you are probably already over budget, he said.

“If I have a $15,000 a month to spend and hit that limit half-way through the month, I can’t just shut down. Cloudability gives us daily reports with our incremental usage spend and detailed cost analysis across multiple accounts and projects,” Fuller told me. That can flag problems before they get out of hand.

Like many AWS users, Fuller loves what he’s able to do with all those AWS services, but the complexity of tracking their usage can be overwhelming. Cloudability’s tool helps him figure out where to use discounted reserved instances vs. other, pricier instance types for example. “If I’m overspending in areas of I/O peformance, I can see that right away and maybe rethink my test model.”

Cloudability and its rivals have to contend with each other but increasingly with the Amazon mothership itself which keeps adding more management and assessemnt tools of its own. But, as Forrester Research analyst Dave Bartoletti said few months ago: “Amazon’s tools will get better and better but Amazon has no desire to get you to use less of its services. It’s like in storage — You’d think EMC would be the best vendor of storage management but historically they haven’t been.”

Keep in mind: All these  third-party vendors rely on Amazon-supplied data to work their magic, and given the rather bracing competition so far, I would expect the others to add similar reporting capabilities pretty quick.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data third-quarter 2012
• Understanding and managing the cost of the cloud

News of an updated version of Trusted Advisor — complete with new features and its free trial (for the month of March) — was unveiled on the AWS blog early Monday morning. Before now, Trusted Advisor was available to customers who signed up for enterprise or business class AWS support.

According to the blog, Trusted Advisor looks over a customer’s AWS environment and makes suggestions on how to save money, boost performance and shutter security gaps:

“Because the AWS Trusted Advisor draws upon the aggregated operational history of hundreds of thousands of AWS customers, you can be confident that the recommendations that it makes can help you to save money, bolster your security profile, improve the fault tolerance of your application, and increase overall performance. This is a unique and powerful benefit that is only possible with cloud-based, API-enabled infrastructure.”

Meanwhile, companies like Newvem, Cloudyn, Cloud Vertical and Cloudability have to be more than a little worried about this new tool, although they’d be the first to tell you that their own respective offerings watch and measure AWS better than Amazon itself does.

A Newvem spokesman characterized the freebie as big news for AWS users and “a great value as a broken-to-fix support play as in something is wrong with my security, I’ll use Trusted Advisor to fix it.” But, he added, Newvem provides more insights on how to improve a user’s AWS resource usage and to evaluate costs, risks and assets. Newvem started charging for its service late last year.

Amazon’s news is a no-brainer for a company that knows it needs to provide more enterprise-class support and monitoring options to placate enterprises used to having such tools, as GigaOM reported last summer. But it also illustrates the issue that, to grow, Amazon is encroaching more and more on spaces pioneered by small members of its ecosystem. Being an AWS technology partner is a risky proposition that is not for the faint-of-heart or the slow-of-foot.

What about the little guys?

Usually, when small companies characterize a huge company’s incursion into their territory as a validation of their strategy, it’s time to pat them on the head and offer condolences. In this case, however, there is some truth that a smaller, more nimble third party (aka Newvem, Cloudyn, et al) can offer more value.

As Forrester Research analyst Dave Bartoletti told me last month with regard to some Cloudyn news:

“Amazon’s tools will get better and better but Amazon has no desire to get you to use less of its services. It’s like in storage — You’d think EMC would be the best vendor of storage management but historically they haven’t been.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Understanding and managing the cost of the cloud
• Cloud and data first-quarter 2013: analysis and outlook

We all know that people are putting tons of their digital stuff into Amazon; The S3 object count hit a trillion in June and it’s gone nowhere but up since.

Cloudyn CEO Sharon Wagner

Sharon Wagner, CEO of Raanana, Israel-based Cloudyn, said his company’s experience monitoring some 400 AWS customers shows surprising things about their usage, many of them relating to storage.

“We found huge inefficiencies. You’d be amazed at the amount of storage that is retired. We slice and dice that and make recommendations that you move that inactive stuff from S3 to Glacier,” he said. Glacier is a lower cost option for archival storage. If you haven’t looked at that document in a year, perhaps its time to ship it off to Glacier and pay less.

If a customer is running versioning on its storage, all those versions do not show up in their AWS console, Wagner said. That means they’re paying for them but don’t necessarily have visibility into them.
Update: An Amazon spokeswoman said via email, that starting in December, the S3 console does allow users to see their versions. And she said that customers can use AWS Lifecycle Policies to automatically move data from S3 to Glacier.

Also, in terms of Glacier it’s worth pointing out that customers can use our Lifecycle Policies to easily and automatically move data from S3 to Glacier.

If you think of Amazon’s various storage tiers, the old analogy holds. Elastic Block Store (EBS) is your active storage — think flash — while S3 is disk and Glacier is tape. Wagner’s argument is you keep your hot storage in EBS, close to your application, move your less active stuff to S3 and all the rest you cart off to Glacier.

Cloudyn also introduced a new service that helps track usage of Amazon’s RDS database and recommend when it’s time for a customer to move from pricier on-demand instances to less costly reserved instances that are typically booked in one- and three-year blocks.

“A database is not the kind of thing that you spin up and down… If a customer has RDS instances they’re not using, we recommend they sell them back to the market. We already do that for EC2 [compute instances.]“

Cloudyn is one of a half dozen or so companies including Cloudability, Newvem and CloudVertical, that are trying to make money helping customers optimize their use of Amazon infrastructure and minimize their spend. The problem is Amazon itself is coming out with more granular tools to help customers monitor its own services. This week it came out with an email alert system for RDS usage, for example. You can elect to get notifications when your database shuts down or restarts, when a backup starts or ends, or when a failover of a multi-zone instance starts or finishes. Amazon also offers its own Cloudwatch service to provide billing alerts etc.

But, as Forrester Research analyst Dave Bartoletti pointed out: “Amazon’s tools will get better and better but Amazon has no desire to get you to use less of its services. It’s like in storage — You’d think EMC would be the best vendor of storage management but historically they haven’t been.”

Bartoletti said this ecosystem of small third parties comes in handy now that many cloud adopters have gotten over the first high of cloud use and need to settle down and get serious about tracking cost and usage. Cloudyn, in his view, has an interesting perspective because of the historical data it keeps about its customers’ use.

This story was updated at 11:19 a.m. PDT with Amazon comment.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data first-quarter 2013: analysis and outlook
• The role of converged infrastructure in the data center
• Cloud and data third-quarter 2012

Newvem CEO Zev Laderman (left) with Amazon CTO Werner Vogels

Large enterprises will negotiate their own deals, but for smaller accounts the company will offer free services until the customer goes over 50,000 AWS resource hours per month.  Then it charges a cent or two per additional resource hour depending on usage. According to Newvem’s price list, there are additional fixed-rate charges for more advanced analytics of S3 storage or EC2 reserved instance use.

One service the company is particularly proud of is tracking AWS usage by department or business function within a large organization. “Amazon itself is very horizontal — it’s just infrastructure — they’ll tell you how many instances and how much storage you use all in one bill. What we do is let you slice and dice Amazon resources against your divisions — no one else can do that,” said Newvem president Zev Laderman. (Update: For the record, Cloudability disputes this, saying it offers similar services.) “If you have R&D and engineering and manufacturing folks and want to tie all AWS usage to each unit to make sure they’re consuming it properly, if there are security issues or vulnerabilities with how ports are setup, if you want to charge-back from a central unit, you can do all that.” 

It’s getting crowded in here

There are a dozen or so companies in this AWS stalking market and they all claim to do what the others cannot. Cloudability, which now says it monitors $250 million in cloud spending,  charges customers a percentage of their overall cloud spend after a free 30-day period. Dublin-based CloudVertical offers similar tiering. Cloudyn offers basic services for free, then tiered pricing for more advanced services above that. There will be more third-party analytics and reporting services coming down the pike.

One newbie is New York-based CloudAware, an AWS consultancy that has done monitoring and analytics work for AOL, Sony and Reed Elsevier (see disclosure) and is now looking to sell a service based on that work. CloudAware will provide a service that alerts management, via a news feed-like feature, to any changes in configurations to AWS resources, said CloudAware CEO Mikhail Malamud. “If someone launches a firewall change and that change is on a non-standard port that will kick off an approval request,” Malamud said.

Mission: Making AWS less inscrutable

These Amazon partners all fulfill a need by enterprise customers to get a better grasp on what many see as a black box.  For one thing, Amazon services are billed as cheap and easy, but cheapness is all relative. At least one Amazon expert experienced unexpectedly high charges  moving S3 data to Amazon’s cheaper Glacier data archiving service for example. So there is a real need for services that monitor and alert customers about their AWS usage.

Nuances aside, these companies all compete with each other. But probably more worrisome longer term for  the startups in this space is that Amazon itself keeps adding deeper and broader billing and analytics capabilities. Offerings like its Trusted Advisor service, now in beta and more granular billing, are eating away at territory these small companies have staked out.

The next year or so will, for sure, be an exciting ride for partners in the Amazon ecosystem.

Disclosure: Reed Elsevier, the parent company of science publisher Elsevier, is an investor in Giga Omni Media, the company that publishes GigaOM.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Cloud and data third-quarter 2012
• Understanding and managing the cost of the cloud

Cloudyn CEO Sharon Wagner.

The latest partnership is between Cloudyn, which captures and analyzes info about user’s AWS utilization — cost, performance and utilization of EC2 instances etc.  – and Scalr which helps users’ deploy and manage their cloud over time. Under a new pact, customers can see Cloudyn’s breakdown of their AWS usage on their Scalr dashboard  and get recommendations on a course of action. Once they choose an option, Scalr will automate it in one click, Sharon Wagner, CEO of Cloudyn told me via email.

Scalr itself supports Amazon, Eucalyptus, Cloudstack, Nimbula and Rackspaceclouds but the initial joint offering is for AWS only.

Newvem, another company that watches AWS usage for customers announced a slate of partnerships at AWS: Reinvent two weeks ago. Its partners include Cedexis, Cloudant, Datapipe, Garantia Data and others.

Other startups in this cloud monitoring field also include Cloud Vertical and on Wednesday, Apptio joined the fray with a free Apptio Cloud Express for tracking AWS, Windows Azure and Rackspace workloads and making optimization recommendations.

All of these companies hope to carve off a piece of a growing market. Gartner estimates that enterprise spending on public cloud services will grow to $207 billion in 2016 from $109 billion this year. But cloud watchers will have to duke it out not only with each other but with the cloud providers themselves as they add more analytics to their own services. Amazon, for example, is beta testing a Trusted Advisor service which provides many of the same metrics as these third parties.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• In Q3, Big Data Meant Big Dollars
• Understanding and managing the cost of the cloud

As of Thursday, Newvem is making its analytics broadly available, just a few days after Cloudability did the same.  Newvem said it’s gained valuable insights from the 8 months its free service has been in trial.  First and foremost “we’ve learned that cloud users are sick and tired of hearing about saving costs … we’re helping cloud users [instead] focus on their business and how the cloud impacts profit,” said Cameron Peron,  Newvem VP of marketing.

The Israeli company measures all Amazon resources used by a given company in real time and uses that data to recommend better deployment options.  A dashboard exposes key performance indicators (KPIs) for C-level executives, IT managers and developers, Peron said.

This market for analytics and dashboards into what some call Amazon’s “black box” is getting pretty crowded.  Not only are there a raft of feisty startups, but established players like Rightscale are starting to get into the act with their own tools. But the biggest factor on the horizon may be Amazon itself  which is starting to make its own costs more trackable.

If you’re confused about which of these tools to use, all of these companies, including Newvem, will be at Amazon’s ReInvent show later this month in Las Vegas. You can check them out for yourself and also hear what Amazon CEO Jeff Bezos and CTO Werner Vogels have to say. At Structure Europe last month, Vogels promised all sorts of “unbelievable” new services to come from the company some of these are bound to be announced at the show.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Understanding and managing the cost of the cloud
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

The Portland, Oregon-based startup says its new analytics provide a picture of actual AWS use and generates reports of actual use, recommending actions to save money.

“We focus on continuous IT improvement — the ability to enable cloud users to put IT where they want it,” company co-founder and CEO Mat Ellis told me in an interview. “As a company, we know who’s wasting money because we track the costs — we take the bills.”

The biggest problem with true cost assessment is finding out what developers and others are actually doing in Amazon. Cloudability’s solution is to push company finance chiefs  to deny any spending that happens outside Cloudability’s reporting structure. If you buy X number of EC2 instances and this much S3 storage on your expense account, you won’t be reimbursed for it unless you use Cloudability.  (That sounds good in theory but I’d like to see it in practice. Developers are notoriously hard to wrangle.)

Cloudability Pro analytics  offers a dozen or so configurable reports and allows custom views of infrastructure use using more than 30 metrics

Because Amazon offers such a wide — and ever-changing — array of services with shifting prices, getting a handle on projected as well as actual cost is difficult . A raft of companies — Newvem, Cloudyn, UptimeCloud — have sprung up to ease this load while incumbents like Rightscale are adding similar cost and workload assessing tools. Even Amazon itself is taking a stab at making its costs more trackable and predictable.

One longtime AWS user speaking at an AWS Meetup in Cambridge, Mass. last night said buying into Amazon Reserved Instances — the priciest types — incurs its own risk. When you lock into a Reserved Instance — for one or three years — you get a lower per hour rate, but you may not use the whole period. Amazon may lower the price during your allotted period, or it may roll out a better instance type in that period. Still, given the relatively low cost compared to deploying servers in house, he doesn’t see much of a downside.

Tools like Cloudability claim they’ll let you wring the most possible value of whatever instance types you buy. Unlike many of the other vendors mentioned, Cloudability says it will, at some point, offer its super-duper reporting and analytics across all the major clouds, not just Amazon. However right now, the tool only works with Amazon.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: Cloud and big data woo enterprises
• Understanding and managing the cost of the cloud
• Cloud computing 2012: a pessimist’s guide

Check out the rest of our Structure Europe live coverage here, and a video recording of the session follows below.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Cloud and data third-quarter 2012
• Understanding and managing the cost of the cloud
• Infrastructure Q1: Cloud and big data woo enterprises