Key Criteria for Evaluating Data Science Solutionsv2.0

An Evaluation Guide for Technology Decision-Makers

Table of Contents

  1. Summary
  2. Data Science Primer
  3. Report Methodology
  4. Decision Criteria Analysis
  5. Evaluation Metrics
  6. Key Criteria: Impact Analysis
  7. Analysts’ Take
  8. About Andrew Brust

1. Summary

Data science solutions are a prerequisite for devising and implementing predictive and prescriptive analytics at enterprise levels of sustainability, security, and governance. The discipline is evolving to incorporate aspects of non-statistical artificial intelligence (AI) techniques including symbolic reasoning, rules-based systems, and semantic knowledge graphs. The bulk of data science offerings focus on classic and advanced machine learning (ML) algorithms, some of which function at the compute resource scale of deep learning. Often, the user base for these solutions consists of data scientists with backgrounds in statistics and computer science. ML engineers specializing in putting models into production and monitoring them frequently use these resources too.

Many aspects of data science solutions are designed to help organizations deal with the scarcity of these valued professionals—particularly data scientists. There’s an enterprise need to liberate data scientists from manual, ad hoc work (like data preparation) that monopolizes their time and distracts them from creating innovative business solutions. Traditionally, such grunt work was performed in siloed environments specific to individual data scientists. Thus, it often suffered from a lack of consistency, code reuse, data governance, and collaboration.

Data science solutions were devised to streamline these processes, making them repeatable, more transparent, and more productive than the previous disjointed efforts were. These solutions service the full spectrum of the data science lifecycle, which includes data preparation, model training, feature engineering, testing, and deployment. There’s also a regular monitoring and recalibration cycle for ensuring models operate in production as they were designed to, in notebooks and data science sandboxes.

By integrating with common frameworks used in data science (including tools for coding, ML libraries, and resources for scoring and deploying models), such solutions considerably expedite work in this discipline. They also make the process less siloed while helping it conform to conventional enterprise standards for data governance and security. Consequently, these tools assist organizations in scaling their data science efforts to match the demands of contemporary business needs for AI and its immense data quantity requisites. Granted, the value of open source tooling impacting this space is considerable. However, commercial data science solutions command a significant amount of this space to formalize the varying steps required in this field to enhance its efficiency and solidify its worth.

The assessment of the multitude of vendor offerings on the market for data science takes several factors into account. The most notable of these include the range of employee skills, organizational use cases, attendant applications, budgetary concerns, and mission-critical objectives.

The GigaOm Key Criteria and Radar reports provide an overview of the data science market, identify capabilities (table stakes, key criteria, and emerging technology) and non-functional requirements (evaluation metrics) for selecting a data science solution, and detail vendors and products that excel. These reports give prospective buyers an overview of the top vendors in this sector and will help decision-makers evaluate solutions and decide where to invest.

How to Read this Report

This GigaOm report is one of a series of documents that helps IT organizations assess competing solutions in the context of well-defined features and criteria. For a fuller understanding, consider reviewing the following reports:

Key Criteria report: A detailed market sector analysis that assesses the impact that key product features and criteria have on top-line solution characteristics—such as scalability, performance, and TCO—that drive purchase decisions.

GigaOm Radar report: A forward-looking analysis that plots the relative value and progression of vendor solutions along multiple axes based on strategy and execution. The Radar report includes a breakdown of each vendor’s offering in the sector.

Solution Profile: An in-depth vendor analysis that builds on the framework developed in the Key Criteria and Radar reports to assess a company’s engagement within a technology sector. This analysis includes forward-looking guidance around both strategy and product.