Review of 100 studies finds 91% hide AI math details

A team from the University Medical Center Utrecht published a scoping review in npj Digital Medicine. They examined 100 peer-reviewed publications that used Shapley value explanations for clinical prediction models. These models use patient data, like age and blood pressure, to predict health outcomes.

Shapley values are a method to explain "black-box" AI. They break down a prediction to show which input features contributed most to the result. Imagine a doctor asking why an AI flagged a patient as high-risk. Shapley values list the specific factors, such as high blood pressure, that drove that decision.

The review focused on transparency in healthcare. The researchers found a significant gap in reporting standards. In 91% of the reviewed papers, authors did not disclose the algorithm used to compute these explanations. This makes it difficult for other scientists to verify the results.

There was also a lack of detail regarding the baseline population. In 97% of cases, the source or size of the reference population used for calculations was not mentioned. This reference group is crucial for understanding what the AI considers "normal" or "average."

The study identified four main reasons researchers use these explanations. They include identifying key features, supporting clinical decisions, building trust, and exploratory analysis. However, the authors noted that common misinterpretations exist in the literature.

The team provided practical recommendations for future studies. They emphasized the need for clear reporting of computational methods. This helps ensure that AI explanations in medicine are reliable and reproducible.

This review highlights a need for better documentation in medical AI research. Without knowing exactly how the explanation was calculated, clinicians may place too much trust in the output. The study does not evaluate the clinical accuracy of the models themselves, only the transparency of their explanations.