Comparative Evaluation of Machine Learning Algorithms for Binary Classification of Heart Disease: A Cross-Validated Analysis of a Clinical Dataset

Abstract

Machine learning (ML) algorithms are increasingly used to classify heart disease using routinely collected clinical variables. This study compared nine supervised classifiers to identify the best-performing model for binary heart disease classification. A secondary analysis was conducted using a dataset of 303 observations, 13 predictors, and a binary outcome. Data completeness and outcome distribution were assessed, and no missing values were identified. The dataset included 165 positive cases (54.5%) and 138 negative cases (45.5%). Logistic Regression, Random Forest, Extra Trees, AdaBoost, Gradient Boosting, radial-basis-function Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Gaussian Naive Bayes, and Decision Tree were evaluated using five-fold stratified cross-validation. Numerical predictors were standardized, and categorical predictors were one-hot encoded within preprocessing pipelines. Performance was assessed using accuracy, precision, sensitivity, specificity, F1-score, and receiver operating characteristic area under the curve (ROC-AUC). Logistic Regression achieved the highest reported accuracy (86.5%), precision (85.2%), sensitivity (90.9%), specificity (81.2%), F1-score (88.0%), and ROC-AUC (0.914). Random Forest and AdaBoost achieved ROC-AUC values of 0.901, while Extra Trees achieved 0.896. Decision Tree performed least effectively, with an accuracy of 74.3% and ROC-AUC of 0.739. Logistic Regression emerged as the strongest-performing classifier in this comparison and represents a reasonable baseline for further investigation. However, the findings remain preliminary because confidence intervals, formal paired comparisons, nested hyperparameter tuning, calibration assessment, and independent external validation were unavailable. Further research should reproduce the analysis using the raw dataset and evaluate model generalizability before considering clinical applications.

Introduction

Cardiovascular diseases are a major public health concern, making early identification and risk prediction important for timely medical evaluation and treatment. Machine Learning (ML) techniques can support the prediction of heart disease by analysing demographic, physiological, and clinical data. This study compares nine classification algorithms to identify the best-performing model for heart disease prediction.

The study used a secondary clinical dataset containing 303 observations, 13 predictor variables, and one binary outcome. The predictors included age, sex, blood pressure, cholesterol, maximum heart rate, chest pain type, and other clinical indicators. Data preprocessing involved standardising numerical variables, encoding categorical variables, and incorporating preprocessing steps into the model pipelines to reduce information leakage.

Nine machine learning algorithms were evaluated: Logistic Regression, Random Forest, Extra Trees, AdaBoost, Gradient Boosting, Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Gaussian Naive Bayes, and Decision Tree. Their performance was assessed using stratified five-fold cross-validation and multiple evaluation metrics, including accuracy, precision, sensitivity, specificity, F1-score, and ROC-AUC.

The results showed that Logistic Regression was the best-performing model among the nine classifiers tested, achieving 86.5% accuracy, 85.2% precision, 90.9% sensitivity, 81.2% specificity, an 88.0% F1-score, and a ROC-AUC of 0.914. Random Forest and AdaBoost also performed well, each achieving a ROC-AUC of 0.901. In contrast, Decision Tree recorded the lowest accuracy at 74.3% and ROC-AUC of 0.739.

The findings suggest that a relatively simple and interpretable model can outperform more complex machine learning algorithms on a small clinical dataset. Logistic Regression demonstrated a favourable balance between identifying positive cases and correctly classifying negative cases. However, its predictions included both false negatives and false positives, highlighting the importance of selecting suitable classification thresholds for the intended clinical application.

In conclusion, Logistic Regression appears to be a promising model for heart disease prediction within the dataset examined. Nevertheless, the study has limitations, including its small sample size, lack of external validation, absence of calibration analysis, and lack of formal statistical comparisons between models. Further research using larger, independent clinical datasets, hyperparameter optimisation, calibration assessment, and clinical utility analysis is necessary before the model can be considered suitable for practical healthcare applications. The model should support, rather than replace, professional medical judgment.

Conclusion

Logistic Regression demonstrated the most favourable reported performance among the nine evaluated classifiers, with accuracy of 86.5%, sensitivity of 90.9%, specificity of 81.2%, F1-score of 88.0%, and ROC-AUC of 0.914. Random Forest and AdaBoost were also promising alternatives, while Decision Tree showed the lowest performance. Logistic Regression is therefore a reasonable leading baseline for further investigation using this dataset. Nevertheless, the limited sample size, absence of external validation, lack of uncertainty estimates, and absence of calibration analyses limit the conclusions. The findings are preliminary and should not be interpreted as evidence that the model is ready for clinical use.

References

[1] Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi:10.1148/radiology.143.1.7063747. [2] Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378. [3] Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research. 2011;12:2825–2830. [4] UCI Machine Learning Repository. Heart Disease [Internet]. Irvine (CA): University of California, Irvine. Available from: https://archive.ics.uci.edu/dataset/45/heart+disease. Accessed 13 October 2026. [5] Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters. 2006;27(8):861–874. doi:10.1016/j.patrec.2005.10.010. [6] Farhadi Z, Feizi-Derakhshi MR, Bevrani H, Kim W, Ijaz MF. ERDeR: The combination of statistical shrinkage methods and ensemble approaches to improve the performance of deep regression. IEEE Access. 2024 Feb 21;12:33361-83. [7] Kilimci ZH. Ensemble regression-based gold price (XAU/USD) prediction. Journal of Emerging Computer Technologies. 2022 Jul 7;2(1):7-12. [8] Fong KS, Motani M. SyREC: A Symbolic-Regression-Based Ensemble Combiner. In2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI) 2024 Oct 28 (pp. 774-781). IEEE. [9] Da Silva PM, Lima MN, Soares WL, Silva IR, Fagundes RA, de Souza FF. Ensemble regression models applied to dropout in higher education. In2019 8th Brazilian conference on intelligent systems (BRACIS) 2019 Oct 15 (pp. 120-125). IEEE. [10] Flores MG, Barone MC, Bachur MR, Lin PH. Errors of medical interpretation and their potential clinical consequences: a comparison of professional versus ad hoc versus no interpreters. Annals of emergency medicine. 2012 Nov 1;60(5):545-53. [11] Flores MG, Barone MC, Bachur MR, Lin PH. Errors of medical interpretation and their potential clinical consequences: a comparison of professional versus ad hoc versus no interpreters. Annals of emergency medicine. 2012 Nov 1;60(5):545-53.

Copyright

Copyright © 2026 Taniya Mondal, Durjodhan Mondal. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.