One of the Deadliest Cancers in America May Have Met Its Match

In the following excerpt, Derek talks to Dr. Ajit Goenka about the new Mayo Clinic study that uses AI to detect pancreatic cancer.

Derek Thompson: Last week, the Mayo Clinic reported that their new AI tool could help specialists detect pancreatic cancer up to three years before a typical diagnosis. So you put it all together, a drug that targets the gene driving most pancreatic cancers, a vaccine that teaches the immune system to recognize the invisible, and third, an AI that can spot the cancer in scans years before any human radiologist would catch it. For the first time in a long time, the deadliest cancer in America looks like something we might actually be able to fight and even to cure.

Today’s guest is Dr. Ajit Goenka, a radiologist at the Mayo Clinic who studies AI imaging and was the lead author of this new study. We talked about his research, why AI seems so good at finding cancer, whether this news is, as some AI stories turn out to be, too good to be true, and what medicine might look like in a world where artificial intelligence can read our bodies better than human doctors can. I’m Derek Thompson. This is Plain English.

Dr. Ajit Goenka, welcome to the show.

Ajit Goenka: Thank you for having me.

Thompson: Why don’t you get us started by telling me, who are you and what do you do?

Goenka: I’m a consultant radiologist and a professor of radiology here at Mayo Clinic in Rochester, Minnesota. What I do is that I try to solve the complex problems for the patients who come to Mayo and otherwise. That’s the 100,000 view of what we do. We can go into the nitty-gritties as we speak along.

Thompson: Great. The first thing I want to talk to you about today is this new, remarkable study of AI, radiology, and pancreatic cancer, which was just published in the journal Gut. Headline finding, tell me what you found.

Goenka: The headline finding is that 85 percent of the patients who develop pancreas cancer, they hear the words “pancreas cancer” at a stage where it is too late for them to do anything about it. What we are trying to do here is that we are trying to flip that equation. And the way we are trying to flip that equation is that we are trying to find those signals, those mathematical signals from the images that can tell us well before time whether or not we can cure a particular patient with pancreas cancer.

AI is just a tool that we are utilizing to solve that problem. Our goal is early detection. And what this study shows is that it is eminently possible. There is certainly a lot more work that we have to do, but I think we are at the base camp now and it’s a matter of time before we start climbing Everest.

Thompson: That’s the headline finding. I would really like to understand what you did, how you found these patients, how you found these scans, and in particular, how you allowed AI to read the scans without cheating. Because there’s been some accusations that these AI-assisted imaging readings involve the AI, like, going online, looking up the patient’s medical records, and saying, “That’s pancreatic cancer” or, “Ope, no pancreatic cancer.” And so you’re actually not testing whether the AI can see well at all; you’re just studying whether the AI has access to the internet.

How did this study work, and how did you make sure that the AI was not cheating?

Goenka: That’s a great question. And you’re absolutely right. That’s a fear that all of us dread, is that we don’t want to come out with a study that does that. No. 1 is that the great thing about being at a place like Mayo Clinic is that we get patient referrals from all over the world. We had about 5,000, 5,500 patients in our archives. We went back in time to find out who had a CT scan that was done for an unrelated indication three months to three years prior to the diagnosis.

You mentioned how we ensured that the AI was not cheating. One of the ways we could have done that or we could have allowed the AI to cheat is that on those CT scans, we could have taken those scans where there was a cancer present, you could see it, but it was missed. We didn’t let that happen. The way we didn’t let that happen is through two things.

We had a few team members of ours, radiologists, who looked at each and every one of those CT scans to confirm that there is no cancer present. That is one way you did that. Second thing we did is that we took the right controls, which means that patients who did not have cancer, we made sure that they were demographically and time-wise comparable to what we had in our test set. So that way, we ensured that the AI did not have the opportunity to, in a way, learn the noise: Oh yeah, that CT scan which is a control looks a little different, so I’m going to use that information to make my prediction. So that’s another way.

Those are the two big ways that one can cheat, and we were very careful and deliberate about it.

Thompson: As I was reading this study, I saw that in scans obtained 18 months before diagnosis, the AI radiology was twice as sensitive as radiologists, and in scans obtained more than 24 months before a diagnosis was made, it was three times more sensitive. To my untrained, not-a-doctor ear, that sounds like AI-assisted radiology in this study was roughly three times better at finding pancreatic cancer than the expert radiologist. Is that a fair conclusion to draw?

Goenka: The short answer is yes. But when it comes to clinical practice—because eventually, we are not doing this for presentations or for media attention. I mean, those things are great. It is great that we are talking about a disease that is long overdue. But if you look into clinical practice, beyond just its ability to find cancer, what is also important is its ability to tell somebody that, no, you don’t have cancer, which is a specificity. All of those things have to be taken into account.

So the short answer is absolutely right, but there are more new answers to it, which I’m happy to go into the details with you.

Thompson: One thing that I always wonder about when it comes to diagnostics and a world in which we’re going to get more and more tools for figuring out the ways in which we are sick or might become sick are these two words of “sensitivity” versus “specificity.” Sensitivity meaning, can you find the positives that are there? And specificity meaning, are you ruling out the negatives? Because what I don’t want is to get a full-body MRI every single year. It finds these 10 little cysts in my body that are never going to become anything and then tells me, “Derek, you have 10 possible developing cancers in your lung, in your gut, in your leg.”

That’s going to ruin my life. It’s going to ruin my life emotionally, financially. I might go through a bunch of tests, so it’s going to ruin me physically. We don’t want that. How well was this test at weeding out the positives from the negatives?

Goenka: That’s an excellent question. To add to the complexity of the terminologies here, in addition to sensitivity and specificity, there is a term called “accuracy” that in a way integrates all of those concepts into one metric. And that metric in this case was about .84, .85, which means 85 percent. When you take both those things into account, it was about 85 percent overall.

Thompson: Is that good?

Goenka: I think that is pretty good, because the reason why it’s important is because you have to compare AI to what is out there. You cannot compare AI, or any tool for that matter, to perfection, because there is no perfection. And right now, if you can see what it was, you have the answer. All of these CT scans were called negative. So in this case, your bar was pretty low to begin with. When you take a bar like that and you go to 85 percent, then I think that’s a monumental accomplishment on the part of this tool.

But having said that, not just sensitivity and specificity, what I think most matters in the clinical arena, in the practice, to address a problem that you mentioned is, what is the pretest likelihood of somebody developing this particular disease? To give you an example: Since the time we published this particular thing, we’ve had dozens, hundreds of queries from concerned patients and family members all over the world. They absolutely just keep on asking those questions.

But here’s the thing: You can have a test that is 99.99 percent sensitive, 99.99 percent specific. But if you apply that test to a patient population where the pretest likelihood of them developing that particular disease is very low, you will still get hundreds and thousands of false positives.

To give you the context, the whole-body MRI that you mentioned is in a way archetypical of that particular problem. Someone like you, who’s young and healthy otherwise, should not be getting that test, because you will inevitably not show anything, but you will end up with a lot of these incidental findings that are going to wreak havoc with your life. Because as an individual, you are concerned. You may be reading the statistics, but you might say, “But what if I’m one of those 1 in 100 where the cyst goes on to develop cancer?”

The message here is that this can be applied only in the right patient population, which in this case would be those individuals over the age of 50 who have got certain risk factors that puts them at a risk of pancreas cancer that is high enough to justify an early-detection paradigm like this.

This excerpt has been edited and condensed.

Host: Derek Thompson
Guest: Dr. Ajit Goenka
Producer: Devon Baroldi
Additional Production Support: Ben Glicksman