Researchers tested Llama-3.1-70B for retrieving patient data from electronic health records

A research team recently published a study in npj Digital Medicine regarding the use of large language models to search through complex patient records. The investigators focused on streamlining how doctors find specific clinical details within years of documentation.

They tested 14 locally hosted, open-source models using a dataset of over 1,600 question-answer pairs derived from Finnish cancer patient files. The models operated entirely offline, ensuring data privacy and security during the retrieval process.

Llama-3.1-70B emerged as the top performer, achieving 95.3% accuracy. The Qwen3-30B-A3B-2507 model also performed well while requiring less memory and running with lower latency. These models proved effective even when prompts did not include predefined answer choices.

• Models were tested against structured patient records.
• Most records were in Finnish, with some English and Latin terms.
• Optimization techniques like 4-bit quantization helped reduce hardware requirements.

While the results are promising, the researchers identified significant limitations. About 2.9% of the AI outputs contained clinically relevant errors. Furthermore, the phrasing of a question could sometimes lead to different medical safety outcomes. Do you trust an automated system to navigate your medical history?