Key Criteria for Evaluating Intelligent Virtual Assistantsv1.0

An Evaluation Guide for Technology Decision Makers

Table of Contents

  1. Summary
  2. Intelligent Virtual Assistants Primer
  3. Report Methodology
  4. Decision Criteria Analysis
  5. Evaluation Factors
  6. Key Criteria: Impact Analysis
  7. Analyst’s Take

1. Summary

Intelligent virtual assistant (IVA) technology has transformed automated telephone response systems, such as those typically used in call centers for the first line of support, as well as other automated query and response handling in other channels, such as web and email.

Unintelligent legacy virtual assistants could only manage a single intent of the end user and were not able to conduct a natural conversation. They were disliked by the public for their limited capability and could drive away customers. The latest generation of IVAs that use artificial intelligence (AI) can separate different intents, handle them in turn, and have a natural conversational ability; overall, these solutions score high in customer satisfaction and can operate efficiently on par with human agents. For end users, an IVA offers a rapid response instead of a long queue waiting to speak with a human agent. As such, the IVA is a win for both end users and businesses.

End users are characterized as having a journey, from the initial query to its resolution. An example of a complex journey would be calling an agent to book a flight, during which the IVA must account for several variables:

  • There could be multiple stages with stopovers.
  • Hotel and car bookings may be included.
  • There may be outward and return bookings and different departure and arrival points.
  • Weather may be a factor, such as the likelihood of snow shutting down an airport.
  • Preferences over ticket class and position of seats must be taken into account.

The IVA agent must navigate the multiple requirements while retaining context, managing changes if the end user needs to modify any details, and taking into account multiple knock-on effects.

Less complex journeys can be handled by chatbots, which involve question-and-answer interactions with minimal continuity or memory between the chatbot responses. In this report, we focus on solutions designed to handle complex end-user journeys.

This GigaOm Key Criteria report details the criteria and evaluation factors for selecting an effective IVA platform. The companion GigaOm Radar report identifies vendors and products that excel in those criteria and metrics. Together, these reports provide an overview of the category and its underlying technology, identify leading IVA offerings, and help decision-makers evaluate these platforms to make a more informed investment decision.

How to Read this Report

This GigaOm report is one of a series of documents that helps IT organizations assess competing solutions in the context of well-defined features and criteria. For a fuller understanding, consider reviewing the following reports:
Key Criteria report: A detailed market sector analysis that assesses the impact that key product features and criteria have on top-line solution characteristics—such as scalability, performance, and TCO—that drive purchase decisions.
GigaOm Radar report: A forward-looking analysis that plots the relative value and progression of vendor solutions along multiple axes based on strategy and execution. The Radar report includes a breakdown of each vendor’s offering in the sector.
Solution Profile: An in-depth vendor analysis that builds on the framework developed in the Key Criteria and Radar reports to assess a company’s engagement within a technology sector. This analysis includes forward-looking guidance around both strategy and product.