Application of machine learning methods for missing data imputation and prediction of diabetes risk

Abstract

Missing data is a near-universal feature of clinical datasets, yet its treatment is often relegated to a fixed preprocessing step that is not itself empirically validated. This study aimed to determine which machine learning regression engine most accurately imputes missing clinical values under different missing-data mechanisms, and whether the choice of imputer affects the accuracy of downstream diabetes risk prediction. Three versions of the same clinical dataset of 4,303 patients (Diabetes is the fourth leading cause of death in the world and one of the most common endocrine disorders. According to studies, Type 2 diabetes kills thousands of people around the world every year and imposes huge costs on societies in the form of surgeries and other treatment programs, as well as controlling complications and disability. Therefore, predicting and early diagnosis of this disease can greatly help governments and patients. This dataset is the output of a Chinese research study conducted in 2018-24. It includes 1304 samples of patients who tested positive for diabetes, and the age of the participants ranges from 21 to 99 years old. The dataset was collected according to the indicators and standards of the World Health Organization, making it a reliable source for building diabetes diagnosis models. Researchers and healthcare professionals can use this dataset to train and test machine learning models to predict and diagnose diabetes in patients each containing 17 clinical and demographic predictors (age, gender, body mass index, blood pressure, glycaemic, lipid, liver and renal measures, and smoking, drinking and family history) and a binary diabetes outcome with a prevalence of approximately 30%, were each subjected to a different missing-data mechanism: Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR). A cross-file ground-truth reconstruction enabled direct, rather than proxy-based, evaluation of imputation accuracy on both synthetic and real missing cells. Decision Tree, Random Forest, and a linear-kernel Support Vector Regressor (SVR) were compared as the regression engine within an iterative (MICE-style) imputer using mean squared error (MSE), mean absolute error (MAE), and R², under 10–40% synthetic missingness and on real missing cells. Decision Tree, Random Forest, and Support Vector Machine (SVM) classifiers were then trained on the imputed datasets (stratified 80/20 train–test split, with 5-fold stratified cross-validation as a robustness check) and evaluated using accuracy, precision, recall (sensitivity), F1-score, and ROC-AUC. No formal statistical significance tests were performed, so differences between models were interpreted descriptively. Random Forest achieved the best average imputation performance in the synthetic experiment (MSE = 76.48, MAE = 4.12, R² = 0.948), narrowly ahead of SVR (R² = 0.947) and clearly ahead of the Decision Tree (R² = 0.934), and its R² on real missing cells ranged from 0.954 to 0.959 across the three mechanisms (MAE 3.45–3.87). Random Forest and SVM classifiers showed very similar performance (ROC-AUC 0.911 and 0.912; accuracy 0.865 and 0.864; sensitivity 0.731 and 0.720; precision 0.806 and 0.809), both outperforming a single Decision Tree (ROC-AUC 0.868; accuracy 0.845; sensitivity 0.714), and performance was stable across the thremissingness mechanisms. These results indicate that a well-chosen, model-based imputer can recover missing clinical values accurately and preserve downstream predictive performance, and that Random Forest is a suitable candidate for both imputation and classification in this dataset. However, the findings rest on a single clinical dataset with a reconstructed ground truth, and external validation in other populations is required before they can be generalised.