AI may help with MS diagnosis, treatment decisions, study finds
Bots do as well as neurologists with disease specialty on 20-question test
Artificial intelligence (AI) platforms did better than most neurologists at answering a 20-question assessment about multiple sclerosis (MS) in a recent study, suggesting that AI may be a helpful tool for MS care.
Neurologists with an MS specialty scored as well, on average, as the AI platforms, while neurology residents with less than two years of training had the lowest scores.
The study, “Artificial Intelligence Versus Neurologists: A Comparative Study on Multiple Sclerosis Expertise,” was published in Clinical Neurology and Neurosurgery.
Test focus: diagnostic criteria, treatment options, managing complications
Large language models, commonly called AI, are computer programs that work by compiling immense amounts of human writing in order to create human-like responses. ChatGPT and Gemini are well-known examples.
As use of AI has become more common in recent years, researchers have been interested in how these tools might improve care for people living with chronic, complex diseases like MS.
In theory, AI could help with a disease diagnosis and in managing MS treatment, especially when patients aren’t able to access expert MS neurologists — but this is only feasible if AI can give accurate answers.
Researchers in Turkey conducted a study in which they gave a 20-question MS test to various AI platforms, as well as to neurologists — 37 neurology specialists, including six with an MS specialty and actively working with patients, and 79 neurology residents (27 with three to five years of training).
The questionnaire included multiple-choice questions with a level of difficulty similar to that of neurology board exams, and it focused on aspects such as diagnostic criteria, drug and nondrug treatment options, and management of disease complications.
“The aim of the study is to assess the accuracy and scope of MS related knowledge, focusing on diagnostic criteria, treatment options and disease management strategies, as tested among neurologists and AI bots,” the researchers wrote.
MS specialists and AI bots, on average, got 17 of 20 questions correct
Results showed that, on average, the neurologists answered 12 out of 20 questions correctly. Neurology residents did notably worse than neurologists, answering only nine questions correctly on average, though residents who had more than two years of training performed similarly to neurologists.
However, “specialists active in MS clinics scored significantly higher” than others in their field, getting an average of nearly 18 questions right for a score of 17.67.
Among all the AI models tested, the average score was 17 out of 20. The best-performing model, called Claude 3.5, answered all the questions correctly except for one — an accuracy level matched by one neurology resident and exceeded by one MS specialist. These data lend credence to the idea that AI may be a useful tool to help guide MS diagnosis and treatment.
“These findings highlight AI’s potential as a valuable clinical decision-support tool, particularly in settings where MS specialists may not be readily available,” the researchers wrote.
They stressed, however, that there’s a big difference between answering a questionnaire and actually managing patients in clinical practice, noting that more work is needed to understand both the strengths and limitations of using AI in MS care.
“The findings suggest AI holds promise in supporting MS diagnosis and treatment, though challenges remain in nuanced cases,” the researchers wrote. “While AI complements neurologists, further studies are essential to understand its potential and limitations.”
About the Author
