Skip to content

Main Navigation

Gigaom Search
  • Menu

  • About Us
  • ResearchExpand
    • Cloud & Infrastructure
    • Data Infrastructure, AI & Analytics
    • Edge & Networking
    • Security & Risk
    • DevOps & Applications
  • Analysts
  • Search
  • Sign in
Become a Client
security-4038043_1920
Image credit: Pete Linforth
Sponsored by

Related Research

GigaOm Radar for Performance Testing Tools Michael Delzer
Kasten by Veeam Enrico Signoretti
Jake Dolezal, William McKnight Sep 17, 2020 (Oct 13, 2020) -- TCO & Benchmark

High Performance Application Security Testing v1.0

Product Evaluation: NGINX App Protect vs. ModSecurity (plus AWS Web Application Firewall)

Table of Contents

  1. Summary
  2. API Security in the Cloud
  3. GigaOm API Workload Test Setup
  4. Test Results
  5. Conclusion
  6. Appendix: Recreating the Test
  7. Disclaimer
  8. About NGINX
  9. About Jake Dolezal
  10. About William McKnight
  11. About GigaOm
  12. Copyright

Summary

Data, web, and application security has evolved dramatically over the past few years. Just as new threats abound, the architecture of applications—how we build and deploy them—has changed. We’ve traded monolithic applications for microservices running in containers and communicating via application programming interfaces (APIs)—and all of it deployed through automated continuous integration/continuous deployment (CI/CD) pipelines. The frameworks we have established to build and deploy applications are optimized for time to market—yet security remains of utmost importance.

The challenge of securing and innovating is profound, and requires a lightweight and integrated security solution that won’t impede performance and delivery. For example, DevOps teams need security controls that work across distributed environments without invasively slowing down or burdening the release cycle. The maturation of these controls and processes ultimately transitions into the realm of DevSecOps, where security is built into the CI/CD pipeline.

The multitude of deployed apps, APIs, and microservices produces a constant flow of communication and data among applications that requires active management—both internal and external. Apps themselves can vary greatly in the protocols, allowed methods, authorization/authentication schemes, and usage patterns. Perhaps most important, IT departments need granular control over the entire application ecosystem to prevent security breaches and attacks, be they man-in-the-middle, distributed denial of service, or script/code/SQL injection attacks.

While security is of utmost importance, the pace of modern business demands high performance, and this is especially true in application- and microservice-enabled enterprises. The conventional approach—deploying a perimeter Web Application Firewall (WAF) to protect applications by filtering and monitoring traffic between the app and the Internet—is no longer enough. Even internal communication between apps and microservices on the trusted corporate network can be compromised and must be addressed. A defense-in-depth strategy is needed with multiple WAFs.

This report focuses on web application security mechanisms deployed in the cloud and closer to your apps. The cloud enables enterprises to differentiate and innovate with microservices at a rapid pace, and allows microservice endpoints to be cloned and scaled in a matter of minutes. The cloud also offers elastic scalability compared to on-premises deployments, enabling faster server deployment and application development and less costly compute. However, the cloud is just as vulnerable, if not more so, to attacks and breaches as on-premises APIs and apps are.

Our focus is specifically on approaches to securing apps, APIs, and microservices that are tuned for high performance and availability. We define “high performance” as companies that experience workloads of more than 1,000 transactions per second (tps) and require a maximum latency below 30 milliseconds across the landscape.

Make no mistake, for many organizations, performance is a big deal—they need to ensure secured transactions at rates that keep pace with the speed of their business. A WAF or application security solution cannot be a performance bottleneck. Many of these companies seek a solution that can load balance across redundant microservices and enable high transaction volumes.

The numbers add up. If a business experiences 1,000 transactions per second, that translates into 3 billion API calls in a month. And it is not uncommon for large companies with high-end traffic levels to experience 10 billion or more API calls in a 30-day period. Make no mistake, performance is a critical factor when choosing an API security solution.

Benchmark Testing

In this report, we performance test three security mechanisms on NGINX: ModSecurity, NGINX App Protect, and AWS Web Application Firewall (WAF). This last product was tested as a fully managed security offering. Note, ModSecurity is commercially distributed by NGINX and will be referred to as “ModSecurity” throughout the rest of this report.

In our benchmarks, NGINX App Protect outperformed ModSecurity at all tested attack rates. NGINX App Protect produced 92% lower latency than NGINX running ModSecurity at the 99th percentile at 1,000 transactions per second (tps) on the 5% bad request test. In our tests, the latencies for App Protect and ModSecurity diverged at the higher percentiles, becoming pronounced at the 95th percentile and above.

For fully managed offerings, NGINX App Protect produced 82% lower latency than AWS WAF at 1,000 tps on the 5% bad request test. Since AWS WAF is fully managed, we do not know what underlying compute resources are working behind the scenes, which makes an apples-to-apples performance comparison difficult. Once again, latency differences were minimal until the 90th percentile, with a significant difference witnessed at the 99th percentile and above.

On a single small 2 CPU and 5.25GB of RAM EC2 instance, we captured the maximum transaction throughput achieved with 100% success (no 5xx or 429 errors) and less than 30ms maximum latency. NGINX App Protect produced about 5,000 requests per second, compared to only 2,000 requests per second with ModSecurity. App Protect provides the same level of throughput as hitting the API directly without a WAF in between.

Testing hardware and software in the cloud is very challenging. Configurations may favor one vendor over another in feature availability, virtual machine processor generations, memory amounts, storage configurations for optimal input/output, network latencies, software and operating system versions, and the workload itself. Even more challenging is testing fully managed, as-a-service offerings where the underlying configurations (processing power, memory, networking, and the like) are unknown. Our testing demonstrates a narrow slice of potential configurations and workloads.

As the sponsor of the report, NGINX opted for a default NGINX installation and API gateway configuration out of the box – the solution was not tuned or altered for performance. GigaOm selected identical hardware configurations for both App Protect and ModSecurity. The fully managed AWS WAF was used “as-is,” since, by virtue of being fully managed, we have no access, visibility, or control over its infrastructure.

We leave the issue of fairness for the reader to determine. We strongly encourage you to look past marketing messages and discern for yourself what is of value. We hope this report is informative and helpful in uncovering some of the challenges and nuances of security architecture selection.

We have provided enough information in the report for anyone to reproduce this test. You are encouraged to compile your own representative workloads and test compatible configurations applicable to your requirements.

Full report available to GigaOm Subscribers.

Subscribe to GigaOm Research
  • Tweet
  • Share
  • Post

Related Research

Fintech Icon  On Abstract Financial Technology Background .
Image credit: monsitj
Sponsored by
Enrico Signoretti Apr 14, 2021 (Apr 14, 2021) -- Solution Brief

Adventist Risk Management Data Protection Infrastructure

Companies always want to enhance their ability to quickly address pressing business needs. Toward that end, they look for new ways to…

Read More
Security & Risk
Group of people watching futuristic GUI.
Jon Collins Apr 13, 2021 -- Blog

The Why of Value Stream Management

If you’re like me and have been around the block in tech more than once, you’ve seen three-letter acronyms come and go.…

Read More
Value Stream Management
network-3396348_1920
Image credit: Pete Linforth
GigaOm Research Apr 12, 2021 -- Blog

Tough Threat Landscape Demands Better Data Protection for Hybrid Cloud

The number of businesses migrating to the cloud is constantly growing. Hybrid cloud is now considered a standard approach by organizations of…

Read More
security-4038043_1920
Image credit: Pete Linforth
Michael Desmond Apr 9, 2021 (Apr 9, 2021) -- Blog

Secure Insight: GigaOm Partners with the CISO Series

Don’t look now, but GigaOm, the analyst firm that enables smart businesses to future-proof their decisions, is forging new partnerships to extend…

Read More
Security & Risk
web-4861612
Image credit: Gerd Altmann
GigaOm Research Apr 9, 2021 -- Blog

The Time is Now for Edge Experimentation

What is an edge platform? In a recent report, GigaOm Radar for Edge Platforms, Analyst Chris Grundemann says the term itself remains…

Read More
futuristic city
Image credit: Jackie Niam
Enrico Signoretti Apr 8, 2021 -- Blog

Data Storage Acceleration

Flash memory is now the standard for storing active data in the data center. NVMe and NVMe over Fiber (NVMe-oF) are on…

Read More
Data Management, Data Storage

Stay on top of emerging trends impacting your industry with updates from our GigaOm Research Community.

Join Research Community
Gigaom
  • About Us
  • Analysts
  • Press Room
  • Contact
  • Twitter
  • Facebook
  • LinkedIn
  • RSS Feed
  • Newsletter
  • Privacy Policy
  • Terms of Service
  • © GigaOm All Rights Reserved 2021
This website uses cookies; by continuing you are a agreeing to our Privacy Policy Accept
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled

This is an necessary category.

Save & Accept