（Written By WANG.Y.X. Translated By GU.X.Y.）Recently, Professor Tien Yin Wong, the Vice Provost ofTsinghua University, Director and Chair Professor of TsinghuaMedicine, and a renowned expert at the Ophthalmology Center of Beijing Tsinghua Changgung Hospital and Beijing Visual Science and Translational Eye Research Institute(BERI), together with Associate Professor WangYaxing , a long-term appointed faculty member at the Ophthalmology Center and BERI, and a team of experts from multiple units, published an important study on thegeneralizability of foundation models based on an Asian-specificdataset.
The results of this study suggest that foundation models face similar challenges to conventional models in generalizability, particularly in terms of data diversity limitations. The study further indicates that ensuring diversity of data sources and promoting global research collaboration are key to addressing this challenge in the current development of healthcare foundation models.
In recent years, foundation models have provided a new path for the development of medical AI. While conventional medical AI model construction usually faces the challenges of heavy data collection workloads and high labeling costs, foundation models are pre-trained on a large amount of unlabeled data, which can be fine-tuned to be applicable to various types of downstream tasks without the need to train from scratch. RETFound, previously published in Nature,is representative of this type of model, however, it remains unclear whether such foundation models are truly applicable to other demographic groups.
The study investigated the applicability of the foundation models in different groups based on Asian population data through three tasks: glaucoma diagnosis, coronary heart disease diagnosis, and stroke risk prediction within three years. The research team selected the RETFound foundation model, which is based on large-scale retinal image pre-training, and the traditional Vision Transformer model, which is pre-trained on ImageNet, for comparison to evaluate the performance of RETFound on the Asian data (as shown in the figure below).
The results showed that RETFound did not outperform the traditional Vision Transformer when fine-tuned using the full dataset, and the difference in performance between the two was not statistically significant (p ≥ 0.2) in glaucoma diagnosis, coronary heart disease diagnosis, or stroke risk prediction. When fine-tuned using only no more than 25% of the entire dataset, RETFound had a slight advantage, with an AUC improvement of up to 0.03, but again not statistically significant.
This study further indicates that despite the overall strong potential of the RETFound model, it still needs to be supported by richer and more diverse training data if it is to achieve the desired results in specific populations and scenarios. Failure to train foundation models on more diverse datasets and test them in a variety of clinical settings will significantly limit theirgeneralizability and applicability to different demographic groups and clinical situations.
This study highlights the importance of global collaboration on foundation model research, to enhance the universality and fairness of AI models by incorporating a more diverse dataset, which will help foundational AI models to play a greater role in various healthcare scenarios, and provide a strong support for the intelligence and precision of global healthcare service.
The study, titled“How Generalizable Are Foundation Models When Applied to Different Demographic Groups and Settings?” was published in NEJM AI (New England Journal of Medicine - Artificial Intelligence) . WangYaxing,Professor WangXiaofei from Beihang University, and ProfessorYih Chung Tham from National University of Singaporeare the co-corresponding authors. And a team from University College Londonalso contributed to the study.
Paper Link: