Competitive Voice Services Reliability Benchmarkv1.0

Table of Contents

  1. Executive Summary
  2. Scenario
  3. Compared Vendors
  4. Environment: Configuration and Setup
  5. Tests
  6. Results
  7. Analyst’s Take
  8. How to Repeat the Test
  9. Report Methodology
  10. About Dana Hernandez

1. Executive Summary

Whether you are looking for a voice solution or a complete unified communications (UC) platform, verbal communication is a critical requirement. Unified communications as a Service (UCaaS) delivers seamless collaboration functionality, blending telephony, messaging, and video capabilities. (Figure 1)

UCaaS platforms are typically easier and more cost-efficient to deploy than on-premises solutions; they enable flexibility for remote working and can trigger workflows in other applications and services across the enterprise. They can improve productivity through more fluid discussion of work items and ease the sharing of assets connected to discussions. High-quality, face-to-face meetings with low latency allow more natural conversations and lead to better business outcomes. Usage analytics helps teams understand the frequency and quality of internal and external interactions and can drive process efficiencies.

This report compares select vendors for the phone services aspects of UCaaS and tests key areas of functionality, ease of use, quality, and uptime performance supported by service level agreements, synthetic testing, and human testing. In addition, service level agreement (SLA) terms were assessed for each vendor. The selected vendors include Microsoft, RingCentral, and Zoom.

Overall, all three vendors’ uptime performance was high at 100% for the test period. All three vendors have compelling features that, for our testers, enhanced their call and meeting experience. Conversely, all three vendors have areas of potential improvement in their feature sets, although some are already on the vendors’ roadmaps. All three vendors had varying levels of SLA terms and potential credits for customers.

We found that Microsoft performed ahead of the competition in several key areas. Synthetic testing of the platform illustrated an Application Performance Index (Apdex) score of 100% satisfied transactions over the test period, alongside 100% uptime. Apdex is an open standard for measuring the performance and behavior of software applications. The other tested vendors fell short in this area.

Our testers rated Microsoft as the top vendor for both ease of use as well as call clarity and quality. They highlighted features such as live captioning and transcriptions of audio, video, and voice mail as excellent. Microsoft’s reporting of call quality also more closely reflected the actual tester experiences and the synthetic testing results. Microsoft SLA terms were the most publicly available and accessible online. It commits to a slightly lower call-quality percentage than other vendors; however, it is willing to credit customers with 100% of their bill.

Figure 1. Integration of Solutions in a Unified Communications Platform

Context

Today, Microsoft Teams provides customers with an SLA that offers service credits if Teams falls below system uptime targets. The three SLA metrics are:

  • Teams core service: Presence, messaging, and online meetings (VoIP calling not included)
  • Calling plans and audio conferencing: Outbound for 1:1 calling, inbound for audio conferencing
  • Voice quality: On Ethernet/IP phones only

Most Microsoft products have a 99.9% SLA metric target except for 99.99% uptime for Teams Phone, excluding planned outages. During Phase 1 of the benchmark, GigaOm conducted research to assess the overall competitiveness of the SLA Voice/PSTN SLA targets versus the key competitors in the market.

Benchmark Focus and Scope
We applied the following Key Criteria for testing:

  • HD/audio/video (emphasis on audio and phone features)
  • Reporting and analytics
  • Meeting controls, collaboration tools, and call routing
  • Ease of use

High Definition (HD) Audio/Video (Emphasis on Audio and Phone Features)

The core capability of UCaaS solutions remains their delivery of high-quality calls and video with good availability. Strong providers can deliver HD quality video in either 720 or 1080p, with low latency/minimal lag, while minimum offerings are at 360p. Audio remains most important to the user experience and service; solutions must provide quality audio and maintain a stable synchronized connection. Enterprises require large numbers of concurrent participants on calls during meetings, with the top vendors supporting over 200 callers. It’s important to note that last-mile delivery between service provider and customer should provide sufficient bandwidth and hardware solution support to not impact the vendor’s audio and video quality.

Reporting and Analytics
Analyzing call duration, stability, and utilization rates by user groups within a unified environment is critical. Reporting and analytics must be able to monitor and provide insights into uptime, call quality, and facility troubleshooting of performance issues. Strong vendors in this key criteria provide real-time data to participants and admins/managers. These reporting and analytics tools enable enterprises to work with their vendor to evaluate whether uptime SLAs are being met and what actions are triggered if they are not. Reporting and analytics can also reveal the extent of internal collaboration, while data related to external calls can provide insight into how an enterprise interacts with its partners and vendors. The tool’s ability to visualize data and monitor the health of a communication system can be key to an enterprise’s success, and ensure users are compliant with internal and external policies.

Meeting Controls and Collaboration Tools (Emphasis on Phone Service Features)
UCaaS solutions enable a range of methods for information to be shared in various formats. Distributed employees need to work on assets in a collaborative way that goes beyond screen sharing. Whiteboards and other design spaces greatly assist in meeting productivity. As information in these meetings may be sensitive, it’s important hosts and admins have sufficient controls to manage interactions. Similarly, controls should exist to limit channel and team groups beyond set audiences, control audiences within meetings, and restrict user groups in an organization that has stringent rules related to local laws or regulatory requirements.

Ease of Use
At the core of any UC solution is how much it promotes user adoption. Solutions must meet the needs of a range of users, from those struggling with technology to those wanting to leverage the advantages of online collaboration. Intuitive tools with strong UX and easy-to-configure local applications on desktop and mobile devices will lead to more successful implementations. A more intuitive solution will make day-to-day management easier. Intuitiveness also facilitates wider adoption of additional features; however, feature-rich but complicated solutions may not yield the desired results within an enterprise. The ease with which other SaaS solutions can be integrated and controlled will assist in getting the most out of a UCaaS solution.

The GigaOm Key Criteria for Evaluating Unified Communications as a Service (UCaaS) Solutions and Radar for Unified Communications as a Service (UCaaS) reports provide an overview of the unified communications and voice over IP market. They identify capabilities (table stakes, key criteria, and emerging technology) and evaluation metrics for selecting a unified communications/telecommunication service, and detail vendors and products that excel. These reports give prospective buyers an overview of the top vendors in this sector and help decision makers evaluate solutions and where to invest.