Open source AI model LEME outperformed top systems in ophthalmology tests
Researchers developed a new suite of models that beat established systems like GPT-4o in clinical reasoning tasks for eye health.
Medical researchers recently published details on LEME, a collection of open-weight AI models specifically designed to assist with eye care. The team trained these models on over 200,000 instruction examples and thousands of preference-labeled entries to improve how they handle clinical logic.
In head-to-head testing, LEME showed better performance than seven other AI baselines. When measured for accuracy in text generation, it surpassed GPT-4o by 3.32% on standard benchmarks. The study also compared the models against LLaMA-3 70B and Eye-LLaMA for extracting vision data from medical notes, where LEME achieved significantly higher F1 scores.
LEME received the highest overall ratings from attending clinicians; its completeness rating even surpassed that of the expert-written answers.
Attending doctors evaluated how the AI answered patient questions. They consistently ranked LEME higher than the other models, noting that its responses were often more complete than those drafted by humans.
Could this technology help reduce the workload for eye specialists in busy clinics? The researchers have made all code and datasets public to encourage further validation by the medical community.
Despite these results, the models were only tested on specific curated benchmarks and deidentified patient records. The current study does not confirm how the tools will perform when integrated into live, high-pressure hospital workflows or during actual patient consultations.
Source: npj Digital Medicine