Table of Contents
- Summary
- How We Got Here
- Does AI Need to be Explainable?
- What is an Explanation?
- Are Human Decisions Explainable?
- Types of Explanations
- Attributes of a Good Explanation
- Places Where Explanations are Most Important
- The Argument for XAI
- The Difficulties of XAI
- The Argument Against XAI
- Methods to Achieve Explainability
- Possibility 1 – “It’s a black box, trust us.”
- Possibility 2 – “We can’t explain it, but here are stats about how well it works.”
- Possibility 3 – Surrogate Models
- Possibility 4 – “Here are the inputs we use.”
- Possibility 5 – Partial Explanation
- Possibility 6 – Sensitivity Analysis
- Possibility 7 – Interpretive Explainability
- Possibility 8 – Certification of Models
- Possibility 9 – Or Some Partial Combination of the Factors Above
- Alternatives to Explainability
- Alternative 1 – Keep Models Simple
- Alternative 2 – Only Use AI in Unimportant Arenas
- Alternative 3 – Give Up
- Analysis: Will Consumers Demand XAI?
- Regulation of AI
- About GigaOm
- Copyright
1. Summary
What is XAI?
The simplest kinds of AI don’t even look like AI. These include such simple devices as a cat food dish that refills automatically or a sprinkler system that comes on when your lawn is dry. In these devices, the logic is quite simple. With regard to the food dish, if the collective weight of the food falls below some threshold, the refilling mechanism is triggered.
But for more complex systems, the very kind we are building today, the logic isn’t simple, and in fact might rely on the subtle interplay of thousands of variables. When such systems make decisions that affect people’s lives – such as the denial of a home loan – people want a humanly understandable explanation as to exactly why the AI came to the conclusion that it did. These explanations are what we call explainable AI, or XAI. AIs without explainability are referred to as black boxes, a name evoking data going in one side and results coming out the other side, with complete opacity about what is going on in the black box itself.
Explainability isn’t a product. It is a feature of a system that uses artificial intelligence. However, it’s not a bolt-on feature like running boards on a truck. It is a fundamental design decision that begins with understanding the data that is being used to train AI, then choosing the proper type of decision engine, and finally selecting algorithms that explain decisions after they are made. These three steps are often referred to as data explainability, model explainability, and post hoc explainability. Most of the focus in the AI community has focused on post hoc explainability.
Explainable Artificial Intelligence sounds fairly straight forward until you start asking some basic questions, such as: “What exactly is an explanation?” and “Understandable to which humans?”
How We Got Here
The need for XAI was not a top-of-mind issue for most industry experts until relatively recently. For decades, explainability has been a non-issue because the models and techniques we used were simple enough that a conclusion could be understood with a little dedicated inquiry.
In an expert system, for instance, the exact decision tree that the computer went through would be easily available to anyone who wanted to take the time to reproduce what the computer did. Computer systems, until relatively recently, were simply stand-ins for people. We used them not because they came to decisions substantially better than a human would, but simply to do monotonous computation faster and more reliably than a human.
This type of dominant use recently has changed. Machine learning, which focuses on finding patterns in data, has no particular interest in understanding the data. This evolution has resulted in the creation of models with complexity vastly beyond human understanding. The situation we find ourselves in is akin to a weatherman being asked to explain why a hurricane took a certain path. To be sure, it is a ‘knowable’ thing, at least in theory. Well-understood natural forces are the only factors that govern the hurricane’s trajectory, but practically speaking, no human is capable of explaining exactly why one particular storm took the path it did, except in the most general of terms.
Machine learning techniques tacitly acknowledge this limitation. Programmers don’t generally build systems then try to understand the output; rather they merely test to see if the output is correct. Data sets are often broken up into two halves, a training half and a testing half. Models are developed with the first one, and then that model is run on the second half of the data. If the results from the testing set are consistent with what was observed in the training set, we say that the system works. But never is it a requirement that we understand why the model produces accurate results. Thus, decisions are seldom explainable.
Consider this example. You operate a pool cleaning service in Austin, Texas. When you do a search on Google for “pool cleaning Austin,” you come up #5, whereas your main competitor comes up #4.
What would happen if you found a Google engineer and put a question to them: “Why do I come up #5 but my competitor is #4?” The engineer would likely shrug and answer, “Who knows? We index fifty billion pages, and you want to know why one is #4 and one is #5? There are thousands of factors that go into the rank, many of which are simply correlations that have been observed. And they change constantly, as do the various weightings of them.”
On my podcast Voices in AI, Amir Khosrowshahi, a VP at Intel and the CTO of its AI products, summed up the situation we find ourselves in this way:
“There’s been a high emphasis on performance of machine learning models, and that’s been at the cost of other things, and one of those things is transparency and explainability. I think what’s happening now, is that in the process of building machine learning systems, the machine learning researcher has to understand what they’re doing, such that they can make better models.”
The Argument for XAI
We live lives ever more governed by algorithms. The media we consume is suggested to us by algorithms, as are the places we eat. The products we buy are suggested by algorithms, as are the ads we see. The programs we stream, the movies we go to, and the music we hear, are all heavily driven by algorithms. The temperature our home is kept at may be governed by a smart thermometer. So is the route we take to work, as well as the person we are matched with in dating apps, which resumes we consider for jobs, and a hundred other things in our everyday life.
Some say we shouldn’t worry about the machines “taking over” our lives. It is far more likely that control over our lives will be thrust upon them by us, relieving us of the tedium of making untold numbers of decisions every day, often with little more to go on than our gut as our guide. The entertainer Keith Lowell Jensen captured a bit of this when he said of the book 1984, “What Orwell failed to see was that we’d go out and buy the cameras ourselves and that our biggest fear would be that nobody’s watching.”
Still, many people are torn between two opposing views about the increased dominance of algorithms over our daily lives. On the one hand, the value of the algorithms, and the potential for them to better our lives, is widely acknowledged. But along with that feeling comes a sense that we have lost something along the way… that some amount of agency over our own lives is gone, and by reason of that, the world has become a less understandable place, where every day more decisions are made for us as opposed to by us. From this standpoint, the desire to understand how the decisions that govern our lives are made is completely understandable.
But that desire for control is not the only reason people want AI to be explainable. Layered on top of that is a fundamental distrust of the underlying system and those who operate it. There are twin worries that either the algorithms don’t work correctly, or that their results are being manipulated by those with ulterior motives.
Is some self-proclaimed know-it-all algorithm giving us flawed advice? Maybe. After all, how would we know? Are we being deliberately manipulated by a puppeteer pulling our strings, tricking us into making choices that are not in our best interest, but in theirs? Perhaps. It is certainly possible.
It is these concerns that drive the desire for XAI. They are entirely understandable and reasonable, and they are founded on completely legitimate concerns. AIs do have an increasing impact on our lives. They are imperfect and they are sometimes created to further hidden agendas. But explainability has all kinds of significant, and potentially insurmountable, challenges. These challenges are not part of some conspiracy to keep information from people, but reflect a reality that truly explainable AI is at least quite hard, and perhaps even impossible.
Where does our distrust of AI come from? Other technologies have an impact on our lives, but we trust them. Primarily, this lack of trust is a by-product of the newness of the technology. GPS has reached a point where people will often follow it even if they think they know better. But this reliance occurs because GPS has been in widespread use for two decades, and for the most part, our collective experience with it has allowed it to earn a certain amount of trust. If it regularly failed, and sent people to Portland Oregon instead of Portland Maine, or even worse, drove them into a lake, we would be understandably hesitant to blindly follow it.
Unfortunately, AI is new enough that often times it is buggy. In fact, often we are pleasantly surprised when it does work. And its failures are often both embarrassing and widely reported. Consider these recent, high-profile news stories of purported AI failures:
“Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam” The New York Times, March 19, 2018
“Chinese businesswoman accused of jaywalking after AI camera spots her face on an advert” The Telegraph, November 25, 2018
“Passport robot tells Asian man his eyes are closed” New York Post, December 7, 2016
“Amazon’s Alexa started ordering people dollhouses after hearing its name on TV” The Verge, January 2017.
“Microsoft’s racist chatbot returns with drug-smoking Twitter meltdown” The Guardian, March 30, 2016
“Crime-fighting robot hits, rolls over child at Silicon Valley mall” The Los Angeles Times, July 14, 2016
“This $150 mask beat Face ID on the iPhone X” The Verge, November 13, 2017
“IBM’s Watson supercomputer recommended ‘unsafe and incorrect’ cancer treatments, internal documents show” Stat News, July 25, 2018
“Toddler asks Amazon’s Alexa to play song but gets porn instead” NY Post, December 30, 2016
It doesn’t help that AI failures are fertile territory for blockbuster movies. A few of the many examples include Bladerunner, 2001, Ex Machina, Metropolis and Will Smith’s I, Robot. The situations in these movies are obviously not real data points against AI, but tend to undermine trust in AI due to a human cognitive bias called “reasoning from fictional evidence,” whereby these movies make AI seem less reliable.
In addition to distrust of the technology itself, there is often distrust of the motives of the institutions deploying AI. This is also a common theme of movies. In X-Men: Days of Future Past, a powerful corporation, Trask Industries, makes robots called Sentinels, originally created to kill mutants that then came to hunt all of mankind. In addition, there’s Cyberdyne Systems which built the Terminators, Weyland-Yutani Corporation that is the evil corporation in the Aliens franchise, and even Stark Industries, the good guys in The Avengers franchise, made an AI named Ultron that decided to destroy humanity after being plugged into the Internet for just a few minutes.
But you don’t have to turn to science fiction for stories along these lines. It happens in real life. A few recent headlines will suffice:
“How Artificial Intelligence is Being Misused to Harm Students” Forbes, July 16, 2018
“China reportedly using secret AI system to track Muslims” New York Post, April 16, 2019
“The US Army wants to turn tanks into AI-powered killing machines” Quartz, February 26, 2019
“The rise of the KILLER ROBOTS: Armed machines ‘could guard North Korea border’” Express, August 27, 2017
The Difficulties of AI
So why is XAI difficult? To begin with, not all AI models are hard to explain. Certain models lend themselves to explainability. Others do not. For example, an AI that decides whether you have a cold or the flu might be a simple decision tree, beginning with the question, “Do you have a fever?” After that, it might ask about aches and pains, and a few other variables. If our AIs were this straightforward, explainability wouldn’t be an issue. But AI wouldn’t be all that powerful either.
However, the recent advances that we have made in AI, the kinds that are making everyone so excited by the technology, don’t work this way at all. They rely on taking vast amounts of data and finding patterns in that data. The underlying models are often agnostic to the task being performed. The computer doesn’t know if it is learning how to spot cats or cancer. It simply is trying to solve a problem, which is, “Given this set of data and these known outcomes, how could I have best predicted those outcomes from that data?”
There is never a question of why that particular data produces that outcome. It is simply a fact that it does. Sometimes a narrative explaining the results can be imposed on data, which we will discuss later, but even in this case, it is not necessarily, nor perhaps even likely, that that narrative is true; that it captures the underlying causality of the real world. It simply provides a story on which to hang the correlations that are discovered by the AI.
In The Empire Strikes Back, Yoda famously told Luke that, “There is no ‘try’.” But with AI, it is not a stretch to say that there is no ‘why.’ There simply ‘is.’ This model makes this prediction. Why? Because that’s what follows from the data.
To make matters worse, generally speaking, the more accurate an AI is, the less explainable it is. Simple AI, like those that do rudimentary classification or which are based on decision trees, are usually understandable by humans. Generally, linear models and Boolean rule sets are highly explainable. But as you work up in complexity and accuracy, as is the case with graphical models, ensemble methods, and neural nets, explainability becomes far harder.
The Argument Against XAI
The main argument against XIA runs like this: “If AI is required to be explainable, then we are explicitly limiting the science to merely human-level performance. If a human can understand the model, the human could, in theory, replicate it with pen and paper. This massively shortchanges the technology and shackles an otherwise powerful technology.”
The reasoning continues this way:
This sentiment was succinctly expressed by Pedro Domingos, AI super guru and author of the book The Master Algorithm, when he tweeted in early 2018:
“Starting May 25, the European Union will require algorithms to explain their output, making deep learning illegal.”
Groucho Marx famously resigned from the Friars’ Club with the statement: “I don’t want to belong to any club that would accept me as one of its members.” Likewise, many in the AI world wouldn’t want to use any AI that humans could understand. What would be the point of that?
In addition, the argument is that since advanced AI is inherently unexplainable, a requirement for an explanation will result in a kind of pseudoscience, that is: things that look like explanations but really aren’t. This will give some illusion of explainability but at its core is an assertion that’s simply not true. In fact, they do little more than dupe the unsophisticated into thinking they understand the decisions of the AI.
Critics of explainability point out that much of what helps us in our modern world isn’t explainable either. We didn’t know how aspirin worked when it first came out, nor did we understand how penicillin works, and we still don’t know how acetaminophen stops pain today, nor how general anesthetics work. We don’t know why placebos work, even when people know they are placebos.
The irony of the situation is that we live in a world where AI might suggest a treatment that we ignore because it isn’t explainable, but we might still use a medicine that we don’t understand either.