All of those administrative and metadata and peripheral problems with AI systems... I'm not sure I buy that story. Because all of those things are fixable, slowly but surely. The problem is that the AI isn't worth it yet. Even when it performs at its best, it's not massively outperforming radiologists, which is what it would need to do to motivate everyone to fix the little problems.
The issue is that radiologists are pretty good at their jobs. They are not the big problem in medicine, and AI choosing to compete with those guys is just an example of looking for your car keys under the streetlight. AI is relatively good at image data, so they thought they'd try their hand at radiology; but in practical terms no one needs Google to do that.
The big entry of AI into medicine will come when AI offers a killer app that doctors can't currently do well. It might be very early diagnosis of disease from changes in lifestyle noticed by a phone or wearable. Or micro-keyhole surgery. Or something else I can't even imagine yet. But AI engines at the moment are just like the very earliest steam engines: not actually able to go faster than a horse. Definitely worth investing in as a long-term project, but the answer to the question "why isn't this technology in widespread use?" is not because hospital data isn't tidy enough. It's because we already have horses.
Excellent article, but few people will want to believe it. From a view on the ground interacting with colleagues, the overwhelming belief among medical personnel is that AI is coming and in fact Hinton will be right, while you have old-timers who don't accept EMRs and are skeptical for unknown reasons about AI and say it won't work but really for vague reasons so sort of invalid logic. The problem is the big lie equating deep learning with AI. Deep learning is not AI and never will be. It is incapable of causal reasoning. You can force solutions via brute strength (to wit, self-driving applications and possibly CXR's and Pap smears, etc) but until real AI is developed (eg, my work but I won't give a narcissistic plug for it) you will need human doctors. No one will want to read or believe what you wrote above, but it is true.
There is a term that describes "mechanistic/physical modeling" -- "causality" :)
(Approx 600K years ago give or take few 100K years ago small set of mutations allowing causality to emerge (plus psychosis) in pre-modern humans, something that does *not* exist in chimps or other animals or in deep learning. )
Very thorough and well-considered article, Dan. While it's good to be excited about the potential of AI for radiology, we need to be measured and realistic concerning its present potential.
When the next disruptive breakthrough occurs, we can then ratchet up our expectations. Keep us posted!
The Thailand retinopathy paper and the BMJ review seem to validate the hype. The Thailand paper ran into problems based on serious material constraints. They had poor-quality images because they didn’t have time for a 60 second waiting period between eyes, they couldn’t turn off the lights in the imaging room because it was shared with other services, or because the camera equipment was broken(!). A western ophthalmology clinic would not have these issues.
The BMJ article says that 2/36 models are better than a single radiologist for routine mammogram evaluations! The market is young and full of failures, but there are AI products that already can eat a good slice of the radiology pie. To capture most of the market, It doesn’t need to be better than the best radiologists, only the median.
Excellent post! I think I am more bullish than you given the rapid progress on LLMs, especially when combined with foundation models and vision, I think you might get something approaching a model of the human body, but you've convinced me that radiology is a lot harder to automate than I thought.
Excellent.. as a radiologist working on AI,, I find this one of the most honest pieces on this topic I've read in a while. As an example, it takes me a glance to detect tumor in liver.. But the best deep learning models are up to only 60% accurate(roughly speaking) and that is with the deepest networks I can train on latest GPUs.. I am beginning to wonder whether deep learning may have peaked already and we ought to find modern methods
All of those administrative and metadata and peripheral problems with AI systems... I'm not sure I buy that story. Because all of those things are fixable, slowly but surely. The problem is that the AI isn't worth it yet. Even when it performs at its best, it's not massively outperforming radiologists, which is what it would need to do to motivate everyone to fix the little problems.
The issue is that radiologists are pretty good at their jobs. They are not the big problem in medicine, and AI choosing to compete with those guys is just an example of looking for your car keys under the streetlight. AI is relatively good at image data, so they thought they'd try their hand at radiology; but in practical terms no one needs Google to do that.
The big entry of AI into medicine will come when AI offers a killer app that doctors can't currently do well. It might be very early diagnosis of disease from changes in lifestyle noticed by a phone or wearable. Or micro-keyhole surgery. Or something else I can't even imagine yet. But AI engines at the moment are just like the very earliest steam engines: not actually able to go faster than a horse. Definitely worth investing in as a long-term project, but the answer to the question "why isn't this technology in widespread use?" is not because hospital data isn't tidy enough. It's because we already have horses.
Excellent article, but few people will want to believe it. From a view on the ground interacting with colleagues, the overwhelming belief among medical personnel is that AI is coming and in fact Hinton will be right, while you have old-timers who don't accept EMRs and are skeptical for unknown reasons about AI and say it won't work but really for vague reasons so sort of invalid logic. The problem is the big lie equating deep learning with AI. Deep learning is not AI and never will be. It is incapable of causal reasoning. You can force solutions via brute strength (to wit, self-driving applications and possibly CXR's and Pap smears, etc) but until real AI is developed (eg, my work but I won't give a narcissistic plug for it) you will need human doctors. No one will want to read or believe what you wrote above, but it is true.
There is a term that describes "mechanistic/physical modeling" -- "causality" :)
(Approx 600K years ago give or take few 100K years ago small set of mutations allowing causality to emerge (plus psychosis) in pre-modern humans, something that does *not* exist in chimps or other animals or in deep learning. )
Very thorough and well-considered article, Dan. While it's good to be excited about the potential of AI for radiology, we need to be measured and realistic concerning its present potential.
When the next disruptive breakthrough occurs, we can then ratchet up our expectations. Keep us posted!
The Thailand retinopathy paper and the BMJ review seem to validate the hype. The Thailand paper ran into problems based on serious material constraints. They had poor-quality images because they didn’t have time for a 60 second waiting period between eyes, they couldn’t turn off the lights in the imaging room because it was shared with other services, or because the camera equipment was broken(!). A western ophthalmology clinic would not have these issues.
The BMJ article says that 2/36 models are better than a single radiologist for routine mammogram evaluations! The market is young and full of failures, but there are AI products that already can eat a good slice of the radiology pie. To capture most of the market, It doesn’t need to be better than the best radiologists, only the median.
Excellent post! I think I am more bullish than you given the rapid progress on LLMs, especially when combined with foundation models and vision, I think you might get something approaching a model of the human body, but you've convinced me that radiology is a lot harder to automate than I thought.
Excellent post!
Excellent.. as a radiologist working on AI,, I find this one of the most honest pieces on this topic I've read in a while. As an example, it takes me a glance to detect tumor in liver.. But the best deep learning models are up to only 60% accurate(roughly speaking) and that is with the deepest networks I can train on latest GPUs.. I am beginning to wonder whether deep learning may have peaked already and we ought to find modern methods