Researchers say AI prescribes better treatment than doctors

50 Comments

A pair of Indiana University researchers has found that a pair of predictive modeling techniques can make significantly better decisions about patients’ treatments than can doctors acting alone. How much better? They claim a better than 50 percent reduction in costs and more than 40 percent better patient outcomes.

The idea behind the research, carried out by Casey Bennett and Kris Hauser, is simple and gets to the core of why so many people care so much about data in the first place: If doctors can consider what’s actually happening and likely to happen instead of relying on intuition, they should be able to make better decisions.

In order to prove out their hypothesis, the researchers worked with “clinical data, demographics and other information on over 6,700 patients who had major clinical depression diagnoses, of which about 65 to 70 percent had co-occurring chronic physical disorders like diabetes, hypertension and cardiovascular disease.” They built a model using Markov decision processes — which predict the probabilities of future events based on those immediately preceding them — and dynamic decision networks — which extend the Markov processes by considering the specific features of those events in order to determine the probabilities. Essentially, their model considers the specifics of a patient’s current state and then determines the best action to effect the best possible outcome.

Credit: Indiana University

Credit: Indiana University

Specifically, Bennett and Hauser found via a simulation of 500 random cases that their model decreased the cost per unit of outcome change to $189 from the $497 without it, an improvement of 58.5 percent. They found their original model improved patient outcomes by nearly 35 percent, but that tweaking a few parameters could bring that number to 41.9 percent.

It’s not surprising that anyone would think computers and data analysis could be a boon for the health care system:

Furthermore, at Structure: Data on March 20, I’ll be discussing the marriage of big data and health care with Aetna innovation head Michael Palmer.

However, no one (or very few people, at least) suggests that Watson or any computer model can or should replace physicians’ judgment. What they can do, though, is digest quantities of research and case studies that no single human being could, meaning they can take a lot more information into account when computing possible outcomes and treatments.

So, although we won’t hear “Paging Dr. Watson” at the hospital anytime soon, there’s an increasingly high chance our doctors will retire to their offices with our charts and ask a computer system of some sort what might be wrong with us and how they might best fix it.

Feature image courtesy of Shutterstock user Luis Louro.

50 Comments

Tom Lawford MD

For todays new problem physician visit, there are two intertwined data inputs at work. The physician opens with “so whats the problem that brought you here today” and then expands on what he/she has been offered with clarifying questions. Based on what the physical shows and the verbal data the physician gathers, heshe forms a differential diagnosis in mind and orders lab/scans on all the items in the DfDx. This works well enough if the patient is one of the 90% that have “one of the top ten” dignoses. But the next 4% of patients are in “one of the top 100 diagnoses” and this begins to require many more questions and even some intuition to add extras to the lab/scans being ordered. The next 4% of patients are more exotic, and in one of “the top 500” diagnoses, gets harder. The last 2% of patients are in one of the “Top 1000 diagnoses” – very hard. I have confidence in the computer doing a great job on the “top 10 of DX”. But for the most difficult 2% of patients, I am dubious that the computer will notice tiny items such as “the sclera of the eyes has a barely noticeable slight blue blendedin with itsusual white (DX=PXE)”, or that the patient has a face that”somehow has the slight hint of a lions face”, or that the “lower ear lobe comes straight down into the skin and doesnt go up a bit (cardiac problem)” Here I think that the computer would blow it just like that last three specialists blew it.

Galactus

incompetence will be on the rise. Doctors will be spineless morons without he ability to make a decision without looking at the computer for reassurance and the right way. I’ve seen the lack of confidence caused by having the ultimate machine right next to you. You second guess everything you do, because you know it can do it better than you.

Dave Mittman

I find this so interesting.
As a clinician the vast majority of people who come in have straightforward problems. Most do not require Sherlock Holmes or the Mayo Brothers to diagnose and treat them. Medicine is usually straightforward as is treatment and when its not, you go to others you trust which could include computers. What counts so much more is time spent gathering that information. Is the patient telling you the truth? Did you do a full exam and take the TIME to REALLY listen. If you did, you will have a happy patient who knows you care about them. A computer can’t see tell tale bruises and bring up spouse abuse, nor can it say, “How long have you had that lesion on your forehead?” It can’t hug you or figure out that you are not taking your medications because you can’t afford them because you got laid off three months ago. Or say, your Dad always worried too much-how ids your anxiety lately?
I take my medical care from humans, aided by all means to make us better. Give me someone who has time to see me when I need them, is well trained (PAs and NPs included) and takes time to care.
Dave Mittman, PA, DFAAPA
Physician Assistant/Associate
That’s why a good clinician will always be needed.

owenl1998

Hugely significant difference in wording from original to this article – changed ‘could’ to ‘can’ which is the same as saying ‘might’ versus ‘will’. Basically the researchers modeled some stuff out against a database and believe that their technique could have the outcomes listed. But they don’t know that yet

Derrick Harris

Thanks for the comment. At the risk of getting into an argument over semantics, I’d say “can” is accurate. At least in their study — which, yes, is just a study — the model *did* perform better on average than doctors.

Curt Welch

What I think we will see in the future, is on line services that we use to track our own personal medical history over our life times instead of leaving that job to our doctors. Ignoring the fact that many would be concerned about privacy, we could upload everything that happens to us in our life time that would be medically relevant. We could have bathroom scales with internet connections that would update our weight (and height) on line every time we stepped on it. We could take our blood pressure and temperature at regular times so our record has a good history of how our health has changed over the years.

Every cold or flu we came down with could be recorded. Every headache, or unusual ache or pain or any other minor symptom could be recorded. All medication use could be recorded and tracked – even the non-prescription medications. All medical test results could be entered. Family history could be entered.

And of course other major illnesses and diagnoses from doctors could be entered.

Because this would be our personal private history, we wold not need to fear (as much) being “honest” such as including more accurate things we would rather not admit to a doctor, like drug use, alcohol, sex with horses :) etc.

Even where we live, and were we have traveled to recently could be tracked by our phone uploading location data to our private medical record.

All this data is exactly what these big data application need to make high quality diagnoses. Small things a doctor might not have access to, such a recent weight loss, could be an important factor in diagnosing an unusual condition.

Doctors are forced to make a diagnoses based on what he sees in the chart, what he sees sitting in front of him, and 5 minutes of history and exam. Unless you have had a long history with the same doctor, they will remember very little about your history and will be forced to diagnose with only the small amount of data they have in front of them. The big data computer application could crunch your entire life medical history in making the diagnoses, and could, for example, cross reference issues with other patients on the same system to look for correlations – like spotting a tracking a contagious disease outbreak.

With all that to work with, the system could provide highly accurate diagnosis, make recommendations about what tests were needed and justified, and recommend different treatment options. We could take all that information to our doctor, so they can use that as additional input to work with, and then prescribe tests, make a diagnoses, and prescribe treatments. The doctor would have better information to work with, and we would be able to double check the quality of the service we are receiving from our health care providers.

David Lane

Shouldn’t a computer be used to make the determination that computers can diagnose better than humans? These two human conclusions could be completely flawed, guys!

goog

At the end of the day, most doctors are simply diagnosing/treating based on what they’ve read, or been taught. There’s a stubbornness in the medical community, where a lot of doctors refuse to use computers to make diagnoses from a patient presenting a set of symptoms. It’s just their pride – at the end of the day, a lot of what a doctor does is simply memory and pattern recognition; stuff which computers are even better at.

andrewbgross

I am certain that computer AIs of this nature can improve outcomes, but this particular bit of research doesn’t persuade me AT ALL. They didn’t apply their model to any real patients, they just “simulated” 500 patients: made up symptoms, made up actual problems. The patients were simulated using the exact same data in their database that went into programming the AI, so, yeah, of course it’s going to match pretty well. Totally unconvincing.

DanaBlankenhorn

No offense, but GE was developing something like this several years ago, using clinical data from Intermountain Health, which has lower costs than other systems (because it has control of income as well as outgo, therefore an incentive to limit the latter).

But here’s a question you should have asked, Derrick. Will this software be offered open source? Will it be on Github? Or are they looking for a commercial contract that will lead to its being buried?

JA

Hi Derrick Harris, cool research to highlight. Don’t be a jerk and make it sound like doctors are stupid or obsolete though just to generate ‘controversy’ and web hits. That’s very simple-minded and it’s not where the field is going at all. ;)

Andreas Ramos

My father was a surgeon with a 50-year career in medicine. He thought highly of evidence-based decision making in medicine. He often said most of the doctors only used treatments that their fellow doctors used, without any followup on whether it worked. When nurses, ambulance crews, or cops (or their families) got injured in his city, they told the ambulance “Take me to the hospital where Dr. Ramos is.” Would I trust a robot doctor over 75% of doctors? Yes.

Patrick

If we use computers more and more for diagnostics, then would the role of the “Doctor” fall off sharply? I don’t question the effectiveness of using a massive database to draw comparisons in symptoms and possible treatments, but would that mean doctors become medical technicians? Would they still need to go to college for 8+ years and do years of clinicals? Would medical school still be as involved and expensive? I liken it to automotive diagnostics, where computers run a gamut of tests and then direct the technician where to go. Experience still dictates that the technician deviate from the computer when common sense prevails, such as replacing an entire seat mount when only one 12 V seat motor burns out……that’s a complete rip-off. However, can you say that automotive technicians and mechanics are as “skilled” as they were perhaps 20-30 years ago? Does a reliance on computers to point the way dilute the talent pool? Would we have more and more “doctors” getting their medical degree from bogus colleges and online classes because, after all, the computer does the work? Basically, how will this affect the standard we see presently?

Curt Welch

In time, all humans will be replaced by machines. They will be so much better than humans, we won’t trust humans to do the work anymore, whether it’s a doctor, lawyer, chef, limo driver, financial adviser or psychologist, auto mechanic, police officer, or legislator. It’s likely this total replacement will happen long before the end of this century.

But before we get there, these smart tools will augment the humans as they do now. Yes, doctors are likely to become weaker diagnosticians in the process, just like kids that grew up using calculators to do math, and Google to find answers, are not as good at doing math in their head, and don’t spend as much time trying to memorize facts as the past generations did.

But overall, the overall quality/cost ratios of service seems to generally rise, not fall, as these transitions happen.

JNR

Citation for original article:

Bennett, C.C. & Hauser, K. “Artificial intelligence framework for simulating clinical decision making: A Markov decision process approah.” ARTIFICIAL INTELLIGENCE IN MEDICINE. (2012)

DOI: 10.1016/j.artmed.2012.12.003

Ed

Not once in your article do you mention or spell out what AI means. For those stumbling upon these type of articles, it would be helpful to have a description of the abreviations used in the title.

If you are writing just for those that are well versed in the subject, then never mind.

But is always nice to use it, whether artificial or innate, when thinking of you target audience.

Gareth Andrews

There are a few issues here.

The first–and obvious–is that the reason the computer model works is that it’s working with probability based on a larger set of data than the data that any one physician has in her head.

The other side of that is also–hopefully–obvious: the model–or the physician–needs to be alert to the subtle clues that suggest the case before him is not one that should be subject to being viewed as in the majority. I won’t say that Intuition is the key here, but…experience and knowledge of the patient might be. Yes, ideally these factors would be built into the computer model, but history suggests the model will not be complex enough.

Another issue that people who are focused on the point of outcomes are likely to view as unimportant is the issue of Privacy. I don’t think I have to explain what that issue is, but maybe I do. Once a patient’s data is in a computer database, judging from what else has gone on with personal data in every other aspect of Society, don’t bet on it remaining private…and/or not getting into Government hands, for example.

Don’t say it’s not relevant.

Marius

And who will assume responsibility and pay the bills if the system fails to prescribe a correct treatment? The programmers or the software company?

Derrick Harris

This is a fair question, I think, but the answer is probably that the hospital pays (or doesn’t) just like when a doctor misdiagnoses something. Without getting into the legal rationale, there’s also the fact that these systems (and this one is just a model devised for a studY) are being marketed as doctors’ assistants rather than as replacements.

bben

Also too many doctors rely on a quickie diagnosis and a cookie cutter response – A quack nearly had me rushed into the OR for a heart operation. He had me scheduled for a pacemaker implant. One of the nurses whispered that I should get a second opinion. – It turned out I had heartburn. And that particular doctor had performed a near record of pacemaker implants. It turns out that he was known by the staff for doing unneeded heart operations. But the ‘System’ was fixed so the patients didn’t know any of that.

Illy

I should also add, those misdiagnoses could have been resolved with proper analysis of the data at hand. I had a doctor that did not treat a stricture despite it being quite evident. The doctor was too wrapped up in his ego to care and too busy accusing me of drug seeking when in fact I needed surgery for an extremely painful condition.

The ERs in the world also need all the help they can get. A computer giving a preliminary diagnosis would go a long way in getting people the help they need quickly. Or at least, getting them some relief while the doctors / computers sort out their health issues.

Illy

I’ve had too many doctors misdiagnose serious conditions on me, including Crohn’s disease as well as a punctured lung.

coffeecat92

Maybe some physicians and doctors are purposefully prescribing incorrect/costly treatment so that they earn more money.

Reaz Mohmed, E.I.T

They are a business after all. They are not in it to lose repeat customer; I think they prefer the trial and error approach to one that is more precise and provides an improved successful treatment. Isn’t it odd that Forbes published an article on why it is not a good idea for physicians to be rated on the level of care they provide ?

Curt Welch

Even better than giving doctors access to these technologies is giving us patients access to it so we can double check the work of the doctors.

Chris

I am a little confused by this article. So, these guys build a system to predict behavior and decision. Then, they used a simulation to test their own system. Isn’t that a bit self serving? This model of theirs isn’t actually used to treat patients, just simulated patients….I might be wrong, but somehow, I think a human body doesn’t always react the same as a simulation.

Frankly, I think this is a pretty invalid comparison. Until such a day a program can be actually used to make all the clinical judgement instead of a doctor, you simply cannot compare the effectiveness of either.

EEguy

This article doesn’t give enough information to conclude that it’s an invalid comparison. I don’t know what they did, but I can give a little information about how some machine learning systems are trained. What is often done (and likely done in this case), is they compiled a large amount of real patient data, and then split it into a training set and validation set. The training set is used to train (duh) the network, and then the validation set is used to judge the systems efficacy. So they never had to create fake people and mock illnesses, they could have easily used real information, that they network had never seen.

David Mackey

Looking forward to what this can do to reduce healthcare costs in the future.

Josh

You know whats really funny? We could have rolled this technology out years ago with the proper allocation of funds.

BongBong

Let the private sector “roll” it out.

My interest in being monitored and probed by a government robot is even less than what the IRS, the Census or other 3-letter government agencies do now.

Half Pint

My recent experience would have been faster and better diagnosed by a computer. Too many egos and not enough listening got in the way of a diagnosis. Two separate doctors told me I didn’t have what lab tests proved I did. They might have also been helped by Kaizen. Not enough asking why.

Slayerwulfe

i am concerned that in some ways we R still in the dark ages. in the future we will trust tech. more, even with R lives. it has no ego it makes no mistakes outside of programming, lasers R utilized, scanning R bodies with ultra sound etc.because we appreciate & trust the precision of R tech. and we trust in it every day.

Dănuț Haiduc

The R programming language. It’s popular among data scientists.

Slayerwulfe

isn’t a physicians judgement just that a compilation of data that leads to diagnostics & a decision. i don’t want to hear any person tell me they can process information faster than a computerized data base. the best physicians of the future will be robotic and the future should be happening as soon as possible.
slayerwulfe cave

32knots

You can imagine a world where robots are emergency room docs and surgeons? That level of sophistication in a robot to be able to see a bleed suture it etc… I would be extremely impressed if this was feasible in the next 20 years.

Chris Maness

In the next 20 years regular physicians will be using these tools more and more to make intelligent decisions about treatments. There’s nothing that can replace a human intuition and there’s nothing that rival the learning capacity of a good machine learning algorithm. These two are a natural fit for each other. In the next 20 years the importance of the actual doctor diagnose will decrease. Also in the 20 years we will see a huge decrease in the number of people who aggregate information as their occupation (such as the lowly paralegal will be a thing of the past).

Chris Maness

In the next 20 years regular physicians will be using these tools more and more to make intelligent decisions about treatments. There’s nothing that can replace a human intuition and there’s nothing that rival the learning capacity of a good machine learning algorithm. These two are a natural fit for each other. In the next 20 years the importance of the actual doctor diagnose will decrease. Also in the 20 years we will see a huge decrease in the number of people who aggregate information as their occupation (such as the lowly paralegal will be a thing of the past).

Joe

Hi SlayerWulfe.

I agree with you on the notion that a physician’s judgment must, of course, be based on data. However, you should know that while computers are very good at some tasks (such as recursive calculation) they are very poor at others (e.g., pattern recognition) owing to the nature of how computers work (they are very good at doing one thing; one at a time). Computers have a hard time with multiple constraint satisfaction (again, this is why pattern recognition eludes computers) which I imagine is one thing physicians, for now, have the upper hand in.

For more information, look into topics pertaining to cognitive science.

Tim

Joe, I am going to have to disagree with you there. There is no inherent reason a reasonably parallel neural network (such as IBMs watson) cannot utilize an ensemble of modern machine learning techniques to perform intelligent pattern recognition much faster than the human mind. Although these algorithms are not as flexible as our native networks, a physician’s job is mostly reliant on Bayesian inference algorithms. Therefore, such systems are actually better suited toward diagnostics than humans in many (though not all) cases.

Reaz Mohmed, E.I.T

Pattern recognition ? In February 2011, IBM proved that computers can also recognize patterns, by demonstrating the Watson Supercomputer, which recognizes patterns in text data to surpass the capabilities of the human mind. Dude … research.

Comments are closed.