Automated Tumor Segmentation in Prostate Cancer MRI Using Hybrid SE-ResNet & Vision Transformer (ViT)

Abstract

One of the most common cancers in men is prostatecancer,andearlydetectionisessentialtobetter treatment results. Because of its high soft-tissue contrast, magneticresonanceimaging(MRI)isfrequentlyusedto detect prostate tumors; however, manual tumor segmentation is laborious and prone to inter-observer variability. This work suggests an automated prostate tumorsegmentationmethodbasedonahybridSqueeze- and-Excitation Residual Network (SE-ResNet) and Vision Transformer (ViT) architecture in order to overcome these drawbacks. The SE-ResNet model effectively extracts discriminative local features from MRI images, while the Vision Transformer captures long-range global spatial dependencies. Combining these complementary models improves segmentation robustnessandaccuracy.Imagepreprocessingmethods areusedtoenhancethe performance,optimizedtraining techniques and MRI quality are used. The proposed method reduces clinical workload, supports faster diagnosis, and improves segmentation reliability. When compared to current segmentation techniques, experimental evaluation using metrics like accuracy, intersection over union (IoU), and dice similarity coefficient shows better performance.

Introduction

The passage reviews advances in AI-based prostate cancer detection and segmentation using MRI imaging, focusing on hybrid deep learning and radiomics approaches.

It explains that traditional methods like biopsy and PSA tests, as well as manual MRI interpretation, are limited by low specificity, subjectivity, and high workload. To address this, modern research uses radiomics, deep learning, and hybrid CNN–Transformer models to improve accuracy and automate detection.

A key proposed approach is a hybrid SE-ResNet + Vision Transformer (ViT) framework:

• SE-ResNet extracts local features (edges, textures, tumor boundaries).
• ViT captures global spatial relationships and long-range dependencies.
• A decoder combines these features to generate accurate pixel-wise tumor segmentation masks.

The system includes preprocessing (normalization, resizing, augmentation), feature extraction, fusion, and evaluation using metrics like Dice Similarity Coefficient (DSC), IoU, and accuracy. MATLAB-based experiments show strong performance with high accuracy and stable convergence.

The literature review highlights related advancements such as radiomics, weakly supervised learning, transformers, Bayesian deep learning, and attention-based segmentation models, all improving prostate MRI analysis but still facing challenges like computational cost, annotation requirements, and generalization.

Conclusion

An efficient image processing framework for the automated detection of prostate tumor in MRI images using MATLAB was proposed in this project. In the proposed method, contrast enhancement, edge detection, frequency-domain processing and feature fusion are fused, which can significantly enhance visualization and discrimination of abnormal tissues. Experimental resultsfromseveralMRIimagesshowedmultilevel consistent and reliable tumor localization with clearly delineated boundaries. By integratingspatial and frequency information, the stability of the proposed method against noise and intensity disturbance is remarkably improved, and better performanceindetectionisobtained.Ingeneral,the proposed system minimizes human intervention, enables early diagnosis and servesas a dependable platform for computer-aided analysis of prostate cancer.Futureworkwouldincludeextensiontodeep learning based segmentation (and quantitative performance measures), to improve accuracy and clinical relevancy.

References

[1] Jemal, R. Siegel, J. Xu, and E. Ward, “Cancer statistics,” CA: A Cancer Journal for Clinicians, vol.60,no. 5,pp.277–300, 2010. [2] H. C. Ahmed et al., “Diagnostic accuracy ofmulti-parametricMRIandTRUSbiopsy in prostate cancer,” The Lancet, vol. 389, no. 10071, pp. 815–822, 2017. [3] H. Pinckaers, W. Bulten, J. van der Laak, and G. Litjens, “Detection of prostate cancer in whole-slide imagesthrough end- to-end training with image-level labels,” IEEE Transactions on Medical Imaging, vol. 40, no. 5, pp. 1413–1424, 2021. [4] J. Xu et al., “Weakly supervised learning forprostatecancerdetectioninwhole-slide pathology images,” Medical Image Analysis, vol. 67, 2021. [5] K. Tiwari et al., “Quantitative analysis of prostate MR images for cancer detection: challenges and opportunities,” Journal of Magnetic Resonance Imaging, vol. 51, no. 1, pp. 123–136, 2020. [6] Y. Zhang and Q. Li, “ProCDet: A new methodforprostatecancerdetectionbased on MR images,” International Journal of ComputerAssistedRadiologyandSurgery, vol. 16, no. 4, pp. 639–649, 2021. [7] H. Suet al., \"State of the Art and Future Opportunities in MRI-Guided Robot- AssistedSurgeryandInterventions,\"inProceedingsoftheIEEE,vol.110,no.7, pp. 968-992, July 2022, doi:10.1109/JPROC.2022.3169146 [8] Y. Liuet al., \"Exploring Uncertainty Measures in Bayesian Deep Attentive Neural Networks for Prostate Zonal Segmentation,\"inIEEEAccess,vol.8,pp. 151817-151828, 2020, doi: 10.1109/ACCESS.2020.3017168 [9] Chaddadet al., \"Deep Radiomic Analysis to Predict Gleason Score in Prostate Cancer,\" in IEEE Access, vol. 8, pp. 167767-167778, 2020, doi: 10.1109/ACCESS.2020.3023902 [10] L. Y. Hung, H. Zheng, Q. Miao, S. S. Raman, D. Terzopoulos and K. Sung, \"CAT-Net: A Cross-Slice Attention Transformer Model for Prostate Zonal Segmentation in MRI,\" in IEEE TransactionsonMedicalImaging,vol.42, no. 1, pp. 291-303, Jan. 2023, doi: 10.1109/TMI.2022.3211764. [11] T. Hassanzadeh, L. G. C. Hamey and K. Ho-Shon,\"ConvolutionalNeuralNetworks for Prostate Magnetic Resonance Image Segmentation,\"inIEEEAccess,vol.7,pp. 36748-36760, 2019, doi: 10.1109/ACCESS.2019.2903284. [12] Y.Yuan,E.Ahn,D.Feng,M.Khadraand J. Kim, \"SSPT-bpMRI: A Self-supervised Pre-training Scheme for Improving ProstateCancer Detectionand Diagnosisin Bi-parametricMRI*,\"202345thAnnual InternationalConferenceoftheIEEE EngineeringinMedicine&BiologySociety (EMBC),Sydney, Australia, 2023,pp.1-4, doi:10.1109/EMBC40787.2023.10340280. [13] Z. Khan, N. Yahya, K. Alsaih, M. I. Al- Hiyali and F. Meriaudeau, \"Recent Automatic Segmentation Algorithms of MRIProstateRegions:AReview,\"inIEEE Access, vol. 9, pp. 97878-97905, 2021, doi:10.1109/ACCESS.2021.3090825. [14] Yan, J. Ding, H. Zhang, K. Tong, B. Hua and S. Shi, \"SEResU-Net for Multimodal Brain Tumor Segmentation,\" in IEEE Access,vol.10,pp.117033-117044,2022, doi: 10.1109/ACCESS.2022.3214309 [15] M.Ali,S.O.Gilani,A.Waris,K.Zafarand M. Jamil, \"Brain Tumour Image SegmentationUsingDeepNetworks,\" inIEEEAccess,vol.8,pp.153589- 153598, 2020, doi: 10.1109/ACCESS.2020.3018160. [16] Stanzioneet al., \"Semi-Automated Image Segmentation of Peri-Prostatic Tissue on MRI and Radiomics Features Stability: A Feasibility Study for Locally Advanced Prostate Cancer Detection,\"2022 IEEE InternationalConferenceonMetrologyfor Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE), Rome, Italy, 2022, pp. 641-645, doi: 10.1109/MetroXRAINE54828.2022.9967607. [17] Z. Sang, C. Li,Y. Xu,Y. Wang, H.Zheng and Y. Guo, \"FCTformer: Fusing ConvolutionalOperationsandTransformer for3D Rectal Tumor Segmentation in MR Images,\" inIEEE Access, vol. 12, pp. 4812-4824, 2024, doi: 10.1109/ACCESS.2024.3349409. [18] S. Balaji, B. Karthik andD. Jose, \"Using a Hybrid System of Coactive Adaptive Neuro-Fuzzy Inference Based on GLCM for Prostate Cancer Categorization,\"2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India, 2024, pp. 1-7, doi: 10.1109/ADICS58448.2024.10533546. [19] Sharma, R. Gupta, R. Deepa, N. G and R. R, \"Advanced Machine Learning Based Prostrate Cancer Detection,\"2025 6th International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 2025, pp. 1432-1438, doi: 10.1109/ICIRCA65293.2025.11089895. [20] Rishabha Malviya; Shivam Rajput; Makarand Vaidya, \"Use of Artificial Intelligence in Imaging for Bone Cancer,\" in Artificial Intelligence for Bone Disorder: Diagnosis and Treatment, Wiley, 2024, pp.219-243, doi: 10.1002/9781394230914.ch9.

Copyright

Copyright © 2026 Gedhagunta Chandraiah, Karanam Ushaswini, Shaik Fathima, Veerapuram Nandhini, Sandumulinti Shaik Shavali5, Sandumulinti, Pydi Ajay. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.