1. Executive Summary

This report format incorporates a field test/benchmark with a white paper-level paper. The body of the paper provides only a summary of testing/results, with the bulk of the process/findings moved to an annex. It therefore makes for a highly readable, punchy, yet detailed paper.

The below can be treated as an overall structure, plus a checklist of possible things to include depending on the report type.

The goal of our study presented in this paper is to objectively uncover whether NetApp is truly positioned to deliver on value propositions to the enterprise. To meet this objective, we designed a field test derived from monitoring, troubleshooting, optimizing, and securing scenarios common to the modern enterprise with, or in the process of, migrating to a hybrid cloud.

This test measured enterprise response to usual and important situations including greedy/degraded applications, underutilized infrastructure, and ransomware simulations.

We found that, in handling applications impacted by shared resource contention, NetApp Cloud Insights would pay off a small company $232,757 per year and a large company over $3.7 million. In terms of underutilized/over-provisioned infrastructure, the gains would be between $314,842 and over $7.8 million. We tested a simulated ransomware event where over 500 files were encrypted with a file extension over a period of 10 minutes. Cloud Secure successfully detected the event after two minutes, took a snapshot of the affected volume, and issued a notification.

2. Monitoring, Troubleshooting, and Optimizing the Hybrid Cloud

As organizations expand their use of external storage into multiple data centers, private clouds, and virtualization services, it can be difficult to have a unified view of the infrastructure footprint and make enterprise-based decisions. Storage management software provides a unified overview of this infrastructure, which eases and facilitates control of the individual arrays. Storage management solutions allow the enterprise to optimize storage use and utilize infrastructure aligned with the workload in terms of function and cost.

Storage management solutions also help manage the essential non-functionals of storage, such as backup and disaster recovery. As data becomes more critical and voluminous, a storage management solution becomes a necessary component of the stack.

NetApp Cloud Insights is a SaaS-based observability tool. Cloud Insights gives visibility into the infrastructure footprint from a single pane of glass. Enterprises have resources in multiple data centers, private clouds, and virtualization services. Cloud Insights gives the ability to quickly spot capacity issues and trends, and look at the performance of the resources across the environment to see if there are any hotspots so workloads can be moved around to relieve pressure and meet KPIs.

Usual and important situations that NetApp Cloud Insights gets involved in include greedy/degraded applications, underutilized resources, and ransomware attacks.

Greedy applications hog resources. They are utilizing more resources than anticipated, putting a strain on their performance, and degrading the service levels of any neighbors sharing resources on the same infrastructure. These applications are common in the enterprise, yet the resulting constrained CPU and memory usage can be very costly, adding up to millions of dollars per year. Likewise, underutilized/over-provisioned resources are also common and expensive. Cloud vendors claim resource elasticity, but that usually only involves scaling up, not down.

It is simply not enough to put up an enterprise storage infrastructure and let it go.

Ransomware is a form of malware that locks the user out and demands a payment to restore access. It is expected that ransomware will cost the world $265 billion in 2031 (link). In 2021, a company was hit by ransomware every 11 seconds. The average ransom is $84,000, downtime costs $840,000, and the average incident lasts 16.2 days.

Tools like NetApp Cloud Insights stop, detect, and prevent the spread of ransomware. NetApp Cloud Insights helps you recover quickly and develop an overall ecosystem data protection strategy for ransomware.

3. NetApp Cloud Insights

Since its inception in the early 1990s, NetApp has been a leader in the data storage hardware industry. However, through aggressive acquisition and increased breadth in its product offering, NetApp has become an emergent innovator in the hybrid cloud management space. NetApp took the front end of its popular data center management software, OnCommand Insight (OCI), and reengineered a new back end built specifically for the hybrid cloud. This product is called Cloud Insights.

NetApp Cloud Insights was designed to give enterprises the deepest visibility into their infrastructure and applications, both on-premises and in the public cloud. Cloud Insights is a monitoring, troubleshooting, and optimizing platform for use with the entire technology stack—as opposed to a disparate, disjointed, and decentralized set of monitoring and troubleshooting tools.

The value propositions of Cloud Insights are many, and they include:

• Diagnose and determine the root cause of problems faster
• Manage resources more effectively and reduce infrastructure waste
• Meet and exceed Service-Level Objectives (SLO) and Service-Level Agreements (SLA) during and after cloud migrations
• Detect ransomware and data theft before it impacts business
• Consolidate and centralize monitoring and troubleshooting tools into one comprehensive platform

4. Testing Scenarios

The field test includes the following scenarios:

• Greedy/degraded application test
• Underutilized infrastructure test
• Ransomware detection test

Greedy/Degraded Application
In our greedy/degraded application test, we developed a greedy and a degraded application scenario. A greedy application is defined as an app hogging resources from other applications on shared infrastructure. This could be CPU and memory, but more often storage and network bandwidth. In particular, greedy storage applications can chew up IOPS bandwidth to the detriment of other applications on a shared storage device. This can lead to the degraded application or an application experiencing degraded performance due to a greedy application consuming the available resources and constraining the impacted application so much that it cannot keep up with its workload.

A primary challenge in diagnosing these issues is that nothing is inherently “broken.” These applications are designed to consume storage performance, and the storage is intended to serve the apps. It’s just when something starts consuming more than its fair share and more than the infrastructure can handle that the degradation happens—but everything is still working “as designed.”

Besides the obvious performance and potential availability downsides of a degraded application, it can be increasingly difficult to diagnose the root cause of the degradation. Imagine the unfolding of events once an application is observed in a degraded state. A user or some other downstream service possibly reports an issue, and right away, technical support or help desk personnel are involved. If their level 1 or level 2 support does not resolve the issue, the team responsible for the degraded application is involved.

From their perspective, once they rule out the possibility that it is an issue germane to their application, the actual cause of the degradation can be any number of possibilities. It could be network bandwidth, and thus, network administration personnel must be involved. It could be a security compromise, and obviously, SecOps staff must be brought online to investigate. Certainly, the problem could be storage-related, especially if the degraded application’s team notices I/O as a possible chokepoint. As you can see, this can result in a lot of finger-pointing and wasting of people’s valuable time—pulling them from their projects and daily activities to put out a fire that may or may not be in their purview.

Figure 1: Hunting Down the Source of Greedy Application Behavior

Time is money, as they say, and wasting precious employee time chasing down issues is exhausting and frustrating. In an age of rapid employee burnout and discontent, the last thing we want is to waste people’s time and cause more undue stress or frustration.

For our test, we created the following scenario to compare the difference between having and not having Cloud Insights:

• Set up two applications on a shared storage device—one greedy application that ultimately causes the degradation of another
• Perform root cause analysis to show how quickly it can be done—discover the greedy application hogging resources and degrading the performance of another

From this test, we measured the following to assess Cloud Insight’s claimed value proposition:

• Time to root-cause identification and problem mitigation
• The number of people involved
• The amount of time wasted finger-pointing and going in circles

While we did not specifically test or measure it, we want to point out how this use case could be further extrapolated and potentially multiplied in value when considering monitoring and troubleshooting containerized infrastructure and applications, such as Kubernetes. Note that with the extra layer of Kubernetes abstraction, the capability to quickly identify and resolve greedy applications (or containers, pods, services, etc.) can significantly save wasted time and effort. Kubernetes takes complexity, scale, and shared infrastructure to another plane within the hybrid cloud. Thus, the case for a solution like Cloud Insights could potentially prove even stronger.

Over-Provisioned/Underutilized Infrastructure
Infrastructure has become increasingly complex, particularly in a hybrid cloud environment. Having infrastructure that sprawls across the cloud and in on-premises data centers makes it very challenging to wrap our arms around and know what we have, what is overutilized, what is underutilized, and how it all ties together. It also makes determining and controlling costs—both in the cloud and on-premises—even more challenging.

We are all aware of the attractiveness of the so-called simplified pricing models of provisioning resources in public clouds like Microsoft Azure, Google Cloud, or Amazon Web Services (AWS). It is a simple marketing message: you only pay for what you use. However, we know this is not exactly the case. If I provision 1 petabyte of cloud block-level storage (not object-level storage, like S3), and I only fill up 500TB of it, I still pay for the full petabyte. At the end of the day, many factors affect cost. For a compute instance or cloud virtual machine, the cost might depend on the number and type of CPUs or GPUs, amount of memory, type and amount of storage, region or availability zone, whether it’s on-demand or reserved, and so forth. The situation is also very complex for storage, and there are many other cloud components and services to consider. There are hundreds of services available across the three largest public clouds, making it very tedious and cumbersome to calculate the cost for all resources or services across our public cloud footprint.

Calculating costs with on-premises infrastructure can be just as difficult, if not more so, than calculating cloud costs. The true cost of on-premises infrastructure includes not only the costs for servers, racks, and storage resources used (capital expenditure), it also includes management costs, associated network costs, and facilities costs (operational expenditure).

Figure 2. Approach to Managing Infrastructure Cost

With all this complexity, we recommend prioritizing a simplified approach to managing infrastructure cost focusing on eliminating waste.

Many FinOps tools tell you what you’ve spent once you’ve already spent it. Automated tools have limitations to certain cloud services and aren’t broadly applicable. In the end the ones actually doing the work and provisioning the resources may not care about cost anyway—since “fast” tends to beat out “cheap” from an engineering point of view—so highlighting overages, waste, and excess resource to engineer is important. It may not be the wasted money that will prompt a change in how infrastructure teams manage their environment, but rather the suggestion that maybe they’re not managing it as efficiently as they could be. Cloud Insights makes saving money easy to do and shows the waste.

If underutilized infrastructure can be identified and reduced, the benefits are multifold:

• Obvious reduction in cloud costs
• Reprovisioning, repurposing, or reduction in underutilized on-premises infrastructure
• Reduction in people’s time and effort to monitor, maintain, and troubleshoot overly extended infrastructure footprints
• Reduction in environmental impact—we all are becoming increasingly aware of the negative impacts our infrastructure glut has had on the environment, use of fossil fuels, and the global energy crisis

For our test, we set up the following scenario to compare the difference between having and not having Cloud Insights:

• Underutilized storage that could be demoted down to a lower/slower/colder tier
• Unused applications that could be demoted down to a lower/cheaper instance type or eliminated
• Redundant snapshots of unused storage

For our test, we measured the following to assess Cloud Insight’s claimed value proposition:

• Cost savings
• Leaner infrastructure footprint

Ransomware and Data Theft
One of the greatest cybersecurity threats in the past few years has been ransomware attacks. One study shows a 41% increase in ransomware-related attacks, while another study found that in 2020, there were over 450,000 reported ransomware attacks worldwide, costing an estimated $25 billion. We also know many more attacks go unreported. These attacks resulted in an average of 16.2 days of downtime and 33% of companies actually paying the ransom demands. Those cases represent attacks from the outside. Other data attacks involve internal employees stealing data, files, or other intellectual property.

To combat these threats, we recommend an enterprise solution that can:

• Detect activity that resembles a ransomware attack or data theft use pattern and stop it before it’s too late
• Use machine learning to recognize the difference between an attack and legitimate activity/fewer false positives
• Ensure corporate compliance by auditing access patterns to data

For our test, we set up the following scenario to compare the difference between having and not having Cloud Insights:

• Simulate a ransomware event
• Use targeted/conditional alerts to notify

It is difficult to put a price tag on the value of preventing ransomware or intellectual property theft. However, we measured the following to assess Cloud Insight’s claimed value proposition:

• Threat detection—that is, the ability to detect ransomware before it impacts the business
• The amount of time it takes from when the ransomware or data theft begins to receive a notification

5. Methods

To build the test scenarios and take the measurements discussed above, GigaOm used the following processes and methods.

NetApp provided GigaOm with an existing NetApp demonstration and lab environment. The lab environment was monitoring a fairly complex environment of 1,800 virtual machines and 1.5PB of storage.

Greedy/Degraded Application
To simulate the greedy/degraded application scenario, we used a MongoDB server as the greedy application and a NGINX server as the degraded application. Both applications used the same shared storage.

We generated a workload on the MongoDB server that spiked its IOPS as high as 2,500 IO/s. This spike in activity quickly degraded our NGINX server and diminished its IOPS to less than 1.0 IO/s, and disk read/write latencies to over 100ms.

Over-Provisioned/Underutilized Infrastructure
To simulate the over-provisioned/underutilized infrastructure, we used a Cloud Insights storage performance dashboard to calculate:

• Overprovisioned CPU
• Abandoned AWS Elastic Block Storage (EBS) volumes that were not attached to any EC2 instance
• AWS EBS currently provisioned as GP2 that could easily be downgraded to ST1 based on its historical (30 day) IOPS and usage
• Abandoned EC2 snapshots of instances that no longer exist

Ransomware and Data Theft Simulation
To simulate ransomware and data theft, we used two scenarios:

• We simulated a person leaving the company suddenly downloading over 5,000 files
• We encrypted over 500 files, resulting in a new file extension

Return on Investment
For all three scenarios, we calculated the delta between not having Cloud Insights and having Cloud Insights. After completing the tests and taking our measurements, we sought to calculate a fair estimate of the return on investment (ROI) of procuring Cloud Insights and the net value we would realize with Cloud Insights.

To calculate the return on investment, we used the following method:

• Time-effort waste measured as the amount of people’s time and effort saved by the tested platform
• Compared to an estimated (but reasonable) amount of people’s time and effort it would take to realize the same value without the platform
• The monthly or hourly costs of the cloud infrastructure footprint that could be reduced by uncovering its underutilization or abandonment through use of the platform
• Compared to leaving that infrastructure untouched and still incurring their charges month after month

The calculation did NOT include other potential values that are more difficult or impossible to measure:

• Downtime costs: These depend on the nature of the degraded application, particularly if it was directly involved in revenue generation.
• Lost opportunity costs: Similar to downtime, a degraded application may be involved in the loss of future opportunity or sales due to its unresolved degradation.
• On-premises infrastructure reduction, repurpose, or elimination: On-premises hardware and software costs can sometimes be more difficult to quantify because they were likely procured upfront with a one-time CapEx.
• Intellectual property or data theft: We may never know exactly how much IP or data theft could cost in future revenue, lost market share, or increased competition.
• Ransomware attacks: The cost of a ransomware event routinely runs into the millions of dollars in ransom paid and downtime, not to mention public relations damages and loss of reputation.

These incalculable values realized by leveraging a comprehensive monitoring, troubleshooting, and optimizing platform could greatly increase the return on investment—particularly if they prevent a major issue, like a successful ransomware attack.

6. Results

The following section reveals the results of our simulation tests and their resulting time-and-effort and infrastructure costs. In each scenario, we have estimated the impact on three different sizes of enterprises: small, medium, and large. While these “T-shirt” sizes are arbitrary, for our estimations, the medium enterprise is twice as big as the small enterprise in the size of their IT department and has a five-times-bigger infrastructure footprint. Likewise, the large enterprise has twice the IT department and a five-times-bigger infrastructure footprint than the medium enterprise.

Greedy/Degraded Application
For the greedy/degraded application scenario, we made a number of assumptions:

• To calculate the value of people’s time, we took the average salary of IT positions from Indeed.com.
• We also increased the salary by a fully-burdened rate of 29.4% from the U.S. Bureau of Labor Statistics.
• For each greedy/degraded application event, we estimated the number and job title of employees involved in determining the root cause of the problem both with and without Cloud Insights.
• We also estimated the number of hours each type of employee would be involved.
• The calculations are based on the greedy and degraded applications event happening twice a month for a year.
• The delta with and without Cloud Insights is the difference in dollar-person-hours times the total number of incidents over one year.

NOTE: We are not suggesting that by having a monitoring, troubleshooting, and optimizing platform, an enterprise can reduce its IT headcount. We are merely illustrating that people’s time is valuable. They are likely contributing to strategic and revenue-generating projects, and taking time to troubleshoot unnecessarily has downstream impacts on corporate objectives and the bottom-line. Using salary and FTE is the only empirical way to quantify this in our study.

Table 1. Incident Stakeholder Cost

With these costs in hand, the following tables show the differences between NOT having Cloud Insights and having Cloud Insights for three sizes of enterprises.

Table 2a. Greedy/Degraded Application at a Small Enterprise Without CloudInsights

Table 2b. Greedy/Degraded Application at a Small Enterprise With CloudInsights

Table 3a. Greedy/Degraded Application at a Medium Enterprise Without CloudInsights

Table 3b. Greedy/Degraded Application at a Medium Enterprise With CloudInsights

Table 4a. Greedy/Degraded Application at a Large Enterprise Without CloudInsights

Table 4b. Greedy/Degraded Application at a Large Enterprise With CloudInsights

As you can see, the cost of time-effort expands dramatically with each incident and as the enterprise grows.

Over-Provisioned/Underutilized Infrastructure
For the over-provisioned/underutilized infrastructure scenario, we made several assumptions:

• We assume the enterprises have 400/2,000/10,000 (small/medium/large, respectively) cloud virtual machines (e.g., EC2 instances, Azure VMs, etc.) with an average of eight vCPUs per instance.
• For storage, we assumed all enterprises were approximately 10% over-provisioned.
• For cloud VM pricing, we assumed the AWS EC2 r5-family on-demand pricing of $0.063/vCPU/hour.
• For cloud block-level and snapshot storage pricing, we used current AWS EBS pricing.

Current in the charts is before NetApp. After is with NetApp.

Table 5a. Underutilized/Over-Provisioned Infrastructure at a Small Enterprise

Table 5b. Underutilized/Over-Provisioned Infrastructure at a Medium Enterprise

Table 5c. Underutilized/Over-Provisioned Infrastructure at a Large Enterprise

Ransomware Simulation

As mentioned previously, we did not estimate the cost of a successful ransomware attack, but instead measured the efficacy of Cloud Secure on Cloud Insights to detect the event as soon as possible.

Response to Unusual Behavior
We tested the event with an internal employee deviating from their normal file access behavior and suddenly downloading over 5,000 files. As you can see in Figure 3 below, Cloud Secure successfully detected the event and issued a notification.

Figure 3. Cloud Secure Detecting Abnormal User Downloading Activity

Encryption With More Than 500 files
We tested the event where more than 500 files were encrypted with a file extension for 10 minutes. As you can see in the figure below, Cloud Secure successfully detected the event after two minutes, took a snapshot of the affected volume, and issued a notification, thereby protecting the data but not disrupting an application if it were determined to be a false positive.

Figure 4. Cloud Secure Detecting Abnormal Encryption Activity

Return On Investment
Finally, we brought all these results together to see when Cloud Insights delivers on its value proposition and how long before the return on investment in the platform is realized.

To perform this calculation, we must first explain the current pricing structure of Cloud Insights. Cloud Insights is priced by Managed Units (MU). Managed Units are defined as:

1 Managed Unit (MU) = 2 Virtual Machines (VM)

1 Managed Unit (MU) = 4 TiB of unformatted external storage capacity or 40 TiB of NetApp StorageGRID unformatted storage capacity

At the time of this writing, Cloud Insights is offered in two tiers:

• Standard – $6 per MU per month
• Premium – $9 per MU per month (which also included Cloud Secure)

NOTE: This pricing is only the public-facing pricing. Volume and term discounts are offered by NetApp..

In the following tables, we calculated the realized value from our greedy/degraded application and over-provisioned infrastructure scenarios and compared it to current Cloud Insights pricing.

Figure 5. Cloud Insights ROI for a Small Enterprise

Figure 6. Cloud Insights ROI for a Medium Enterprise

Figure 7. Cloud Insights ROI for a Large Enterprise

As you can see, Cloud Insights delivers on its value proposition and virtually pays for itself within the first two months of use, no matter the size of the enterprise or whether one selects the Standard or Premium edition. Because the value of detecting and potentially mitigating a ransomware attack is incalculable and invaluable, we would recommend the Premium edition.

7. Conclusion

NetApp Cloud Insights is positioned to deliver high value propositions to the enterprise. Our tests–measuring enterprise response to usual and important situations including greedy/degraded applications, underutilized infrastructure, and ransomware simulations– proved a strong ROI for NetApp Cloud Insights.

We found that, in handling greedy/degraded applications, NetApp Cloud Insights would benefit a small company by $232,757 per year and a large company by over $3.7 million. In terms of underutilized/over-provisioned infrastructure, the gains would be between $314,842 and over $7.8 million. We tested a simulated ransomware event where over 500 files were encrypted over a period of 10 minutes. Cloud Secure successfully detected the event after two minutes, took a snapshot of the affected volume, and issued a notification.

The costs of going without cloud monitoring can quickly accumulate below the radar, though they are becoming more evident to enterprises. It is hoped that, with these tests, we have brought to light typical enterprise experiences and costs (and risks) with a hybrid cloud infrastructure.

Cloud Insights has the capabilities enterprises need to monitor and ensure the smooth operation of hybrid and multicloud resources. Cloud Insights delivers ROI through complete visibility of the enterprise’s entire infrastructure footprint.

8. About William McKnight

William McKnight is a former Fortune 50 technology executive and database engineer. An Ernst & Young Entrepreneur of the Year finalist and frequent best practices judge, he helps enterprise clients with action plans, architectures, strategies, and technology tools to manage information.

Currently, William is an analyst for GigaOm Research who takes corporate information and turns it into a bottom-line-enhancing asset. He has worked with Dong Energy, France Telecom, Pfizer, Samba Bank, ScotiaBank, Teva Pharmaceuticals, and Verizon, among many others. William focuses on delivering business value and solving business problems utilizing proven approaches in information management.

9. About Jake Dolezal

Jake Dolezal is a contributing analyst at GigaOm. He has two decades of experience in the information management field, with expertise in analytics, data warehousing, master data management, data governance, business intelligence, statistics, data modeling and integration, and visualization. Jake has solved technical problems across a broad range of industries, including healthcare, education, government, manufacturing, engineering, hospitality, and restaurants. He has a doctorate in information management from Syracuse University.

10. About GigaOm

GigaOm provides technical, operational, and business advice for IT’s strategic digital enterprise and business initiatives. Enterprise business leaders, CIOs, and technology organizations partner with GigaOm for practical, actionable, strategic, and visionary advice for modernizing and transforming their business. GigaOm’s advice empowers enterprises to successfully compete in an increasingly complicated business atmosphere that requires a solid understanding of constantly changing customer demands.

GigaOm works directly with enterprises both inside and outside of the IT organization to apply proven research and methodologies designed to avoid pitfalls and roadblocks while balancing risk and innovation. Research methodologies include but are not limited to adoption and benchmarking surveys, use cases, interviews, ROI/TCO, market landscapes, strategic trends, and technical benchmarks. Our analysts possess 20+ years of experience advising a spectrum of clients from early adopters to mainstream enterprises.

GigaOm’s perspective is that of the unbiased enterprise practitioner. Through this perspective, GigaOm connects with engaged and loyal subscribers on a deep and meaningful level.

11. Copyright

© Knowingly, Inc. 2022 "NetApp Cloud Insights: A GigaOm Benchmark Report" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact sales@gigaom.com.