Reliability Reporting in Kaggle-Based Diabetic Retinopathy AI Studies

Abstract

Public retinal fundus datasets have enabled many machine-learning studies on diabetic retinopathy detection, but high reported performance can be difficult to interpret when validation and reporting practices vary across studies. This structured secondary-data audit examined 30 source-traced study rows involving Kaggle-accessible diabetic-retinopathy datasets or closely related external-validation datasets. For each row, dataset use, model family, task framing, reported performance metrics, evidence-use tier, and nine reliability-reporting criteria were extracted into a reproducible workbook. The audit identified incomplete reporting of several reliability markers: external validation was clearly reported in 9 of 30 rows, confidence intervals in 2 of 30 rows, public code availability in 1 of 30 rows, and explicit leakage-control reporting in 12 of 30 rows. No row reached the high-reliability band under the predefined 0-9 rubric. Reported AUROC/AUC and accuracy values were summarized descriptively only because included rows used heterogeneous datasets, tasks, and validation procedures. The study identifies reporting gaps and supports cautious interpretation of high internal Kaggle performance claims. It does not train a new model, clinically validate a diagnostic system, or conduct a formal diagnostic-performance meta-analysis.

Introduction

This study examines the reporting reliability of machine learning (ML) research for diabetic retinopathy (DR) detection using publicly available Kaggle retinal image datasets, rather than proposing a new diagnostic model. Diabetic retinopathy is a major cause of vision loss, and AI-based fundus image analysis has become an important tool for early screening. However, many studies report high performance using public datasets without adequately demonstrating that their models generalize to real-world clinical settings.

The research investigates how frequently previous DR studies report important reliability measures such as external validation, confidence intervals, statistical testing, code availability, data leakage prevention, class imbalance handling, patient-level data splitting, calibration, and clinical performance metrics. The hypothesis is that studies relying only on internal Kaggle validation tend to report impressive accuracy but have lower overall reliability than studies with stronger validation practices.

A structured secondary-data audit was conducted using 30 source-traced studies based mainly on the APTOS 2019 Blindness Detection and Kaggle EyePACS datasets, along with related external validation datasets such as Messidor, Messidor-2, IDRiD, and DDR. Instead of performing a systematic review or meta-analysis, the study focused on evaluating how well these studies reported validation methods and reliability criteria.

To assess reporting quality, the researchers developed a 0–9 reliability scoring rubric. The rubric awards points for reporting key practices including:

• External validation
• Confidence intervals
• Statistical testing
• Public code availability
• Leakage-control measures
• Class imbalance handling
• Patient/image split description
• Calibration or uncertainty analysis
• Clinical metrics such as sensitivity and specificity

The results reveal significant shortcomings in reporting practices. Among the 30 studies:

• Only 9 studies clearly performed external validation.
• 2 studies reported confidence intervals.
• No study clearly reported statistical testing.
• Only 1 study made its code publicly available.
• Leakage-control methods were clearly described in 12 studies.
• Patient-level data splitting was reported in only 2 studies.
• Calibration or uncertainty analysis appeared in only 1 study.
• Clinical metrics were reported in 13 studies.

The average reliability score was 2.85 out of 9, with most studies falling into the low-reliability category, indicating limited reporting of important validation safeguards. The study emphasizes that a low reliability score does not necessarily mean a poor AI model, but rather insufficient documentation of methods that support confidence in the reported results.

Conclusion

This study presents a reproducible secondary-data audit of Kaggle-based and Kaggle-adjacent diabetic-retinopathy AI studies. Across 30 source-traced rows, reliability-related reporting was inconsistent: external validation, confidence intervals, code availability, calibration analysis, and detailed leakage-control reporting were not uniformly documented. The main conclusion is intentionally cautious: high internal Kaggle performance should be interpreted alongside validation design and reporting completeness. The study does not establish clinical performance of any model, but it provides a practical rubric for reliability-aware evaluation of public-dataset medical-AI claims.

References

[1] National Eye Institute. Diabetic Retinopathy. National Eye Institute. https://www.nei.nih.gov/learn-about-eye-health/eye-conditions-and-diseases/diabetic-retinopathy Accessed: 29 June 2026. [2] Kaggle. APTOS 2019 Blindness Detection. https://www.kaggle.com/c/aptos2019-blindness-detection Accessed: 29 June 2026. [3] Kaggle. Diabetic Retinopathy Detection. https://www.kaggle.com/c/diabetic-retinopathy-detection Accessed: 29 June 2026. [4] Gulshan et al. (2016). Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. https://jamanetwork.com/journals/jama/fullarticle/2588763. [5] Voets, Møllersen, Bongo. (2018). Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. https://arxiv.org/abs/1803.04337. [6] Islam, Hasan, Abdullah. (2018). Deep Learning based Early Detection and Grading of Diabetic Retinopathy Using Retinal Fundus Images. https://arxiv.org/abs/1812.10595. [7] Tymchenko, Marchenko, Spodarets. (2020). Deep Learning Approach to Diabetic Retinopathy Detection. https://arxiv.org/abs/2003.02261. [8] delaPava, Ríos, Rodríguez, Perdomo, González. (2021). A deep learning model for classification of diabetic retinopathy in eye fundus images based on retinal lesion detection. https://arxiv.org/abs/2110.07745. [9] Arrieta Ramos, Perdómo, González. (2022). Deep Semi-Supervised and Self-Supervised Learning for Diabetic Retinopathy Detection. https://arxiv.org/abs/2208.02408. [10] Karkera, Adak, Chattopadhyay, Saqib. (2023). Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review. https://arxiv.org/abs/2301.00973. [11] Jain, Gupta, Singhal. (2024). Diabetic Retinopathy Detection Using Quantum Transfer Learning. https://arxiv.org/abs/2405.01734. [12] Pakdelmoez et al. (2024). Controllable retinal image synthesis using conditional StyleGAN and latent space manipulation for improved diagnosis and grading of diabetic retinopathy. https://arxiv.org/abs/2409.07422. [13] Alam, Roy, Kawsar, Rimi. (2024). Enhancing Transfer Learning for Medical Image Classification with SMOTE: A Comparative Study. https://arxiv.org/abs/2412.20235. [14] Mardianta, Affandy, Supriyanto, Wijaya. (2025). Diabetic Retinopathy Detection Based on Convolutional Neural Networks with SMOTE and CLAHE Techniques Applied to Fundus Images. https://arxiv.org/abs/2504.05696. [15] Ahmed. (2025). Addressing High Class Imbalance in Multi-Class Diabetic Retinopathy Severity Grading with Augmentation and Transfer Learning. https://arxiv.org/abs/2507.17121. [16] Chaturvedi, Gupta, Ninawe, Prasad. (2020). Automated Diabetic Retinopathy Grading using Deep Convolutional Neural Network. https://arxiv.org/abs/2004.06334. [17] Shakibania, Raoufi, Pourafkham, Khotanlou, Mansoorizadeh. (2023). Dual Branch Deep Learning Network for Detection and Stage Grading of Diabetic Retinopathy. https://arxiv.org/abs/2308.09945. [18] Hannan, Mahmood, Qureshi, Ali. (2025). Enhancing Diabetic Retinopathy Classification Accuracy through Dual Attention Mechanism in Deep Learning. https://arxiv.org/abs/2507.19199. [19] Kumar, Aditya, Kumar, Bikku, Thota, Kumar. (2025). Stage Aware Diagnosis of Diabetic Retinopathy via Ordinal Regression. https://arxiv.org/abs/2511.14398. [20] Doshi, Shah. (2026). Bridging the Rural Healthcare Gap: A Cascaded Edge-Cloud Architecture for Automated Retinal Screening. https://arxiv.org/abs/2605.14108. [21] Ahmed. (2025). HOG-CNN: Integrating Histogram of Oriented Gradients with Convolutional Neural Networks for Retinal Image Classification. https://arxiv.org/abs/2507.22274. [22] Hashmi. (2026). Robust Diabetic Retinopathy Grading Using Dual-Resolution Attention-Based Deep Learning with Ordinal Regression. https://arxiv.org/abs/2604.17341. [23] Islam, Jany, Ahmed, Khan. (2025). Balancing Accuracy and Efficiency: CNN Fusion Models for Diabetic Retinopathy Screening. https://arxiv.org/abs/2512.21861. [24] Al-Kamachy, Hassanpour, Choupani. (2024). Classification of Diabetic Retinopathy using Pre-Trained Deep Learning Models. https://arxiv.org/abs/2403.19905. [25] Zhu, Qiu, Lepore, Dumitrascu, Wang. (2022). Self-Supervised Equivariant Regularization Reconciles Multiple Instance Learning: Joint Referable Diabetic Retinopathy Classification and Lesion Segmentation. https://arxiv.org/abs/2210.05946. [26] [26] Mok, Bum, Tai, Choo. (2024). Cross Feature Fusion of Fundus Image and Generated Lesion Map for Referable Diabetic Retinopathy Classification. https://arxiv.org/abs/2411.03618. [27] [27] Kar, Akib, Hasib, Yaser, Azim. (2026). Temporal-Enhanced Interpretable Multi-Modal Prognosis and Risk Stratification Framework for Diabetic Retinopathy (TIMM-ProRS). https://arxiv.org/abs/2601.08240. [28] [28] Joshi, Sharma, Rathod, Sawant, Musale, Kalamkar. (2026). Mobile-Ready Automated Triage of Diabetic Retinopathy Using Digital Fundus Images. https://arxiv.org/abs/2602.21943. [29] [29] Florindo. (2026). Chaos-SSL: An Attention-Based Self-Supervised Learning Framework with Chaotic Transformation for Medical Image Classification. https://arxiv.org/abs/2605.27146. [30] [30] Urooj, Banerjee, Shaikh, Thakur, Gupta. (2025). Single Domain Generalization in Diabetic Retinopathy: A Neuro-Symbolic Learning Approach. https://arxiv.org/abs/2509.02918. [31] [31] Florindo, Ornelas. (2026). Attention-Based Chaotic Self-Supervision for Medical Image Classification. https://arxiv.org/abs/2605.04985. [32] [32] Urooj, Banerjee, Gupta. (2025). NEURO-GUARD: Neuro-Symbolic Generalization and Unbiased Adaptive Routing for Diagnostics: Explainable Medical AI. https://arxiv.org/abs/2512.18177. [33] [33] Sánchez-Gutiérrez et al. (2022). Performance of a deep learning system for detection of referable diabetic retinopathy in real clinical settings. https://arxiv.org/abs/2205.05554.

Copyright

Copyright © 2026 Mudit Sharma. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.