A study led by the University of Cambridge has found that GPT-4, a large language model, is approaching the level of specialist eye doctors in terms of clinical knowledge and reasoning skills. When presented with 87 patient scenarios involving specific eye problems, GPT-4 scored significantly better than unspecialised junior doctors, who have a similar level of specialist eye knowledge as general practitioners.
AI Shows Potential to Improve Healthcare Workflow
While the researchers do not believe that large language models will replace healthcare professionals, they see potential for these AI systems to improve healthcare as part of the clinical workflow. GPT-4 could be useful for providing eye-related advice, diagnosis, and management suggestions in well-controlled contexts, such as triaging patients or in areas where access to specialist healthcare professionals is limited.
“We could realistically deploy AI in triaging patients with eye issues to decide which cases are emergencies that need to be seen by a specialist immediately, which can be seen by a GP, and which don’t need treatment,” said Dr Arun Thirunavukarasu, lead author of the study.
Comparing AI to Practicing Doctors for a Fair Comparison
The researchers designed their study to compare the abilities of AI to practicing doctors, rather than to sets of examination results, to provide a fair comparison. “Doctors aren’t revising for exams for their whole career. We wanted to see how AI fared when pitted against to the on-the-spot knowledge and abilities of practicing doctors,” said Thirunavukarasu.
GPT-4 Outperforms Other Large Language Models
The study also tested other large language models, including GPT-3.5, PaLM2, and LLaMA, with the same set of questions. GPT-4 provided more accurate responses than all of them. GPT-4 powers the online chatbot ChatGPT, which has recently attracted significant attention in medicine for its performance in medical school examinations and providing accurate and empathetic messages to patient queries.
Doctors to Remain in Charge of Patient Care
Despite the potential of AI in healthcare, the researchers believe that doctors will continue to be in charge of patient care. “The most important thing is to empower patients to decide whether they want computer systems to be involved or not. That will be an individual decision for each patient to make,” said Thirunavukarasu.
Rapid Advancements in AI Language Models
The field of artificially intelligent large language models is moving very rapidly, with more advanced models being released since the study was conducted. These newer models may be even closer to the level of expert eye doctors in terms of clinical knowledge and reasoning skills.
