Laptop Displaying the GigaOm Research Portal

Get your Free GigaOm account today.

Access complimentary GigaOm content by signing up for a FREE GigaOm account today — or upgrade to premium for full access to the GigaOm research catalog. Join now and uncover what you’ve been missing!

API and Microservices Management Benchmarkv2.0

Product Evaluation: Kong Enterprise, Apigee X and MuleSoft Anypoint

Table of Contents

  1. Executive Summary
  2. Full Cycle API Management
  3. GigaOm API Workload Test Setup
  4. Test Results
  5. Conclusion
  6. Appendix: Recreating the Test
  7. Disclaimer
  8. About Kong
  9. About William McKnight
  10. About Jake Dolezal

1. Executive Summary

Application programming interfaces, or APIs, are a ubiquitous method and de facto standard of communication among modern information technologies. The information ecosystems within large companies and complex organizations encompass a vast array of applications and systems, many of which have turned to APIs for exchanging data as the glue that holds these heterogeneous artifacts together. APIs have begun to replace older, more cumbersome methods of information sharing with lightweight, loosely-coupled microservices. This change allows organizations to knit together disparate systems and applications without creating technical debt from tight coupling with custom code or proprietary, unwieldy vendor tools.

APIs and microservices also allow companies to create standards and govern the interoperability of applications—both new and old—building modularity. They broaden the scope of data exchange with the outside world, particularly mobile technology, smart devices, and the Internet of Things (IoT), because organizations can share data securely with non-fixed-location consumers and producers of information.

The popularity and proliferation of APIs and microservices have created a need to manage the multitude of services a company relies on—both internally and externally. APIs vary greatly in protocols, methods, authorization/authentication schemes, and usage patterns. Additionally, IT teams need greater control over their hosted APIs, such as rate limiting, quotas, policy enforcement, and user identification, to ensure high availability while preventing abuse and security breaches. APIs have enabled their own economy by allowing the transformation of businesses into a platform (and even a platform into a business). Exposing APIs opens the door to many partners who can co-create and expand the core platform without knowing anything about the underlying technology.

Still, many organizations depend on their apps, APIs, and microservices for high performance and availability. For this report, we define “high performance” as companies that experience workloads of more than 1,000 transactions per second (tps) and need a maximum latency below 30 milliseconds across their landscape. For these organizations, the need for performance is equivalent to the need for management because they rely on these API transaction rates to keep up with the speed of their business operations. For them, an API management solution must not become a performance bottleneck. On the contrary, many of these companies are looking for a solution to load balance across redundant API endpoints and enable high transaction volumes. Imagine a financial institution with 1,000 transactions happening per second—that translates to 86 million API calls in a single 24-hour day! So performance is a critical factor when choosing an API management solution.

This report reveals the results of performance testing we completed on these API and microservices management platforms: Kong Enterprise, Google Cloud Apigee X, and MuleSoft Anypoint Flex Gateway.

In this performance benchmark, Kong came out a clear winner—particularly because of its higher rate of transactions per second. Kong’s maximum transactions per second throughput, achieved with 100% success (no 5xx or 429 errors) and less than 30ms maximum latency, was 54,250. By contrast, Apigee X’s maximum throughput was 1,750, and the highest throughput we saw on a MuleSoft Anypoint Flex Gateway was 1,250 responses per second.

Testing hardware and software in the cloud is very challenging. Configurations may favor one vendor over another in feature availability, virtual machine processor generations, memory amounts, storage configurations for optimal input/output, network latencies, software and operating system versions, and the workload itself. Even more challenging is testing fully managed, as-a-service offerings for which the underlying configurations (processing power, memory, networking, etc.) are unknown. Our testing demonstrates a narrow slice of potential configurations and workloads.

As the report’s sponsor, Kong opted for a default Kong installation and API gateway configuration out-of-the-box—the solution was not tuned or altered for performance. The Anypoint Flex Gateway was also not tuned or altered for performance. The fully managed Apigee X was used “as-is” since, by virtue of being fully managed, we have no access to, visibility into, or control of its respective infrastructure.

We hope this report is informative and helpful in uncovering some of the challenges and nuances of API management platforms.

We have provided enough information in the report for anyone to reproduce this test. You are encouraged to compile your own representative workloads and test compatible configurations applicable to your requirements.