Machine Learning and Deep Learning Approaches for Cancer Histopathology Image Classification

Abstract

Early and accurate detection of cancer, via histopathology, can improve patient outcomes significantly. However, manual analysis of microscopic tissue images is time-consuming and subjective. Machine learning (ML), especially deep learning (DL), provides automated solutions to classify cancer cell images. In this review, we summarize state-of-the-art ML and DL techniques for histopathological image classification. We discuss common public datasets (e.g. BreakHis), preprocessing steps (e.g. stain normalization, data augmentation), and modern model architectures (e.g. CNNs like ResNet, EfficientNet, Vision Transformers). Evaluation typically uses metrics like accuracy, precision, recall, F1-score, and ROC-AUC. For example, a transfer-learning approach with ResNet-50 achieved 92.42% accuracy (AUC 0.9986) on an 8-class breast cancer subtyping task [1]. Other hybrid models have reported accuracies up to 99% on binary classification of breast histology [2]. However, there are challenges with overfitting and interpretability. We will conclude by outlining future directions for cancer image classification.

Introduction

Cancer remains a major global health challenge, with millions of new cases and deaths each year. Accurate cancer diagnosis relies on histopathological examination of biopsy slides, but manual inspection by pathologists is slow, labor-intensive, and prone to human variability. To address these limitations, researchers increasingly use machine learning (ML) and deep learning (DL) for automated analysis of histology images.

Traditional ML methods depend on handcrafted features and typically achieve only moderate accuracy. In contrast, deep convolutional neural networks (CNNs) automatically learn complex patterns from raw images and have become the leading approach in cancer image classification. Studies show that CNNs, especially when pretrained on large datasets (e.g., ResNet, VGG, DenseNet), achieve high accuracy in distinguishing benign from malignant tissues, grading tumors, and predicting cancer subtypes.

Advanced techniques such as hybrid CNN-RNN models, attention mechanisms, and transfer learning further improve performance. Some models reach accuracies above 99% on popular datasets like BreaKHis. Data augmentation, stain normalization, and balanced sampling help prevent overfitting and reduce color variability across laboratories.

Emerging methods like Vision Transformers (ViTs) and CNN-Transformer hybrids show promise but require large annotated datasets.

Despite strong results, challenges remain, including dataset imbalance, overfitting, poor generalization across institutions, and limited interpretability. Explainable AI tools such as Grad-CAM are increasingly used to address the “black box” issue.

Overall, DL—especially CNN-based approaches—has significantly advanced automated cancer histopathology classification, improving speed, consistency, and diagnostic support. Future work must focus on generalization, transparency, and clinical integration.

Conclusion

Deep learning has markedly advanced the automated classification of cancer histopathology images. Current models regularly achieve >90% accuracy on benchmark tasks (binary or multiclass) [1] [2], greatly outperforming earlier ML methods. Key findings include the efficacy of transfer learning (ResNet-50 and EfficientNet) and the benefit of data augmentation and stain normalization. Hybrid architectures (e.g. CNN+RNN) can offer marginal gains in capturing spatial context [19]. Future research should address remaining gaps. We need methods, such as explainable AI, to interpret model decisions (e.g. visualizing salient histological features) to build clinician trust. Multimodal integration is another method. By combining histology with genomic or radiology data (as in TCGA projects), we could improve diagnostic accuracy and subtype prediction. Overall, machine learning stands to transform pathology but requires careful attention to interpretability, ethics, and testing, so artificial intelligence can truly help with cancer diagnoses and improve patient outcomes.

References

[1] A. Desai and R. Mahto, “Multi-class classification of breast cancer subtypes using ResNet architectures on histopathological images,” *J. Imaging*, vol. 11, no. 8, p. 284, 2025. [2] M. Yusoff et al., “Accuracy analysis of deep learning methods in breast cancer classification: a structured review,” *Diagnostics*, vol. 13, no. 4, p. 683, 2023. [3] R. Yan et al., “Breast cancer histopathological image classification using a hybrid deep neural network,” *Methods*, vol. 173, pp. 52–60, Feb. 2020. [4] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “A dataset for breast cancer histopathological image classification,” *IEEE Trans. Biomed. Eng.*, vol. 63, no. 7, pp. 1455–1462, Jul. 2016. [5] T. Gurcan et al., “Histopathological image analysis: A review,” *IEEE Rev. Biomed. Eng.*, vol. 2, pp. 147–171, 2009. [6] M. A. M. Khan et al., “A survey of breast cancer diagnosis using histopathological images,” *Comput. Biol. Med.*, vol. 123, p. 103865, 2020. [7] J. Madabhushi and G. Lee, “Image analysis and machine learning in digital pathology: Challenges and opportunities,” *Med. Image Anal.*, vol. 33, pp. 170–175, 2016. [8] A. Cruz-Roa et al., “Automatic detection of invasive ductal carcinoma in whole-slide images with convolutional neural networks,” *Proc. SPIE Med. Imaging*, vol. 9041, 2014. [9] A. E. Carpenter et al., “CellProfiler: image analysis software for identifying and quantifying cell phenotypes,” *Genome Biol.*, vol. 7, no. 10, p. R100, 2006. [10] A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” *Adv. Neural Inf. Process. Syst.*, vol. 25, pp. 1097–1105, 2012. [11] M. Yusoff et al., “A comprehensive analysis of deep learning applications in breast cancer histopathological image classification,” *Appl. Sci.*, vol. 12, no. 7, p. 3456, 2022. [12] A. Raghu et al., “Transfer learning for medical image classification: a literature review,” *BMC Med. Imaging*, vol. 22, no. 1, p. 73, 2022. [13] F. Spanhol et al., “Breast Cancer Histopathological Database (BreaKHis),” *Lab. VisãoRobótica e Imagem*, Univ. Federal do Paraná, 2016. [14] K. Komura and R. Ishikawa, “Machine learning methods for histopathological image analysis,” *Comput. Struct. Biotechnol. J.*, vol. 16, pp. 34–42, 2018. [15] J. R. Zech et al., “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs,” *PLoS Med.*, vol. 15, no. 11, p. e1002683, 2018. [16] F. A. Spanhol et al., “A dataset for breast cancer histopathological image classification,” *IEEE Trans. Biomed. Eng.*, vol. 63, no. 7, pp. 1455–1462, Jul. 2016. [17] A. Belsare and M. Mushrif, “Histopathological image analysis using image processing techniques: An overview,” *Signal Image Process. Int. J.*, vol. 3, no. 4, pp. 23–36, 2012. [18] C. Petitjean et al., “Evaluation of texture-based approaches to breast cancer histopathological image classification,” *J. Pathol. Inform.*, vol. 8, no. 1, 2017. [19] R. Yan et al., “Breast cancer histopathological image classification using a hybrid deep neural network,” *Methods*, vol. 173, pp. 52–60, 2020. [20] S. K. Zhou et al., “A review of transfer learning in medical imaging,” *Annu. Rev. Biomed. Eng.*, vol. 23, pp. 325–357, 2021. [21] M. Yusoff et al., “Performance comparison of CNN architectures in breast histopathological image classification,” *J. Imaging*, vol. 9, no. 5, p. 115, 2023. [22] J. Veeling et al., “Rotation equivariant CNNs for digital pathology,” *Med. Image Anal.*, vol. 68, p. 101849, 2021. [23] S. Macenko et al., “A method for normalizing histology slides for quantitative analysis,” *Proc. IEEE ISBI*, pp. 1107–1110, 2009. [24] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” *ICLR*, 2021. [25] Z. Liu et al., “Swin Transformer: Hierarchical vision transformer using shifted windows,” *Proc. ICCV*, pp. 10012–10022, 2021. [26] H. Zhang et al., “TransBreastNet: A CNN-transformer hybrid deep learning framework for breast cancer subtype classification,” *Sci. Rep.*, vol. 15, p. 19173, 2025. [27] M. Nawaz et al., “Analysis of deep features for breast cancer histopathological image classification,” *IEEE Access*, vol. 8, pp. 90128–90140, 2020. [28] A. Bukhari et al., “A hybrid deep learning model EfficientNet with GRU for breast cancer classification,” *Sci. Rep.*, vol. 15, p. 930, 2025. [29] P. Smailagic et al., “Hybrid deep learning architectures for histopathology image classification,” *IEEE J. Biomed. Health Inform.*, vol. 27, no. 4, pp. 1342–1355, 2023.

Copyright

Copyright © 2025 Raghav Jhawar, Abhiraj Pal. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.