Dual-Model Deep Learning Framework for Tooth Segmentation and Multi-Condition Detection in Panoramic Radiographs

Abstract

Orthopantomogram (OPG) radiographs are one of the main pillars of dental diagnosis, yet their interpretation is subjective, time consuming, and expensive depending on the radiologist. We suggest a deep learning structure to solve this by adding U-Net-based tooth segmentation and YOLOv8-based object detection to analyze the dental abnormalities in an automated manner. The segmentation model deals with anatomical localization and gives estimates of the regions of focus, and the detection model deals with clinically significant findings like cavities, impacted teeth, fillings and implants. The U-Net has 97.23% accuracy on validation data in the segmentation of teeth and the YOLOv8 model has a total mAP at 0.5 of 78.7% at 640 × 640, with the difference between individual classes rep-resenting imbalances in our limited dataset. One of the design considerations was clinical usability via continuous learning approach. The framework has an interactive course-correction interface where clinicians can manipulate segmentation masks and refine detection bounding boxes. An error-correction loop is built on top of these corrections, where both models are refined at each annotated case, as the number of annotated cases increases. The final output is an automatically generated PDF that displays patient details along with the dental analysis summary. The dental anomaly and the count of each type is displayed. Confidence scores for prediction can also be displayed depend-ing on the user’s choice. This end-to-end pipeline is practically feasible and cost effective while being useful in real-time for clinicians.

Introduction

The research presents a deep learning-based system for automated dental disease detection using Orthopantomogram (OPG) panoramic X-ray images. Dental conditions such as cavities, impacted teeth, fillings, and implants require accurate identification for effective diagnosis and treatment. However, manual interpretation of OPG images can be time-consuming and challenging because of overlapping anatomical structures, unclear boundaries, and metallic distortions.

To address these challenges, the study proposes a dual-model framework combining U-Net for tooth segmentation and YOLOv8 for dental pathology detection. U-Net identifies tooth-bearing regions in the X-ray image, while YOLOv8 detects and classifies dental abnormalities. This combination aims to improve anatomical localization, detection efficiency, and clinical interpretability.

1. Objectives

The main objectives of the proposed system are to:

• Automatically identify tooth regions and dental abnormalities in OPG images.

• Combine tooth segmentation and pathology detection in a unified framework.

• Estimate prediction confidence and identify uncertain or difficult cases.

• Improve model performance through clinician corrections and continuous learning.

• Generate structured PDF reports containing AI findings, confidence scores, and clinical recommendations.

• Provide an accessible, web-based platform for dental image analysis.

2. Literature Review

Previous research has demonstrated the effectiveness of U-Net and its variants for medical image segmentation and YOLO models for real-time object detection. However, many existing approaches treat segmentation and pathology detection as separate tasks. Some systems also struggle with imbalanced datasets, low-contrast features, and complex anatomical structures.

The study addresses these limitations by integrating segmentation and detection with a human-in-the-loop learning mechanism. This allows dental professionals to correct model predictions and helps the system improve through subsequent retraining.

3. Methodology

The proposed system follows a two-stage deep learning approach.

Stage 1: U-Net-based tooth segmentation

U-Net identifies tooth-bearing regions in OPG images and generates binary segmentation masks. Images are resized to 512 × 512 pixels and converted to grayscale. The model uses an encoder–decoder architecture with skip connections and is trained using Binary Cross-Entropy and Dice loss functions.

The segmentation dataset contains approximately 1,200–1,500 OPG images with corresponding masks.

Stage 2: YOLOv8-based pathology detection

YOLOv8 detects four dental categories:

• Cavities

• Impacted teeth

• Fillings

• Implants

Images are resized to 640 × 640 pixels, and the model predicts bounding boxes, class labels, and confidence scores. The detection dataset contains 780 annotated OPG images.

The two models provide complementary information about anatomical regions and potential dental abnormalities. The workflow also includes preprocessing, connected-component analysis, visualization, and report generation.

4. Continuous Learning and Model Improvement

An important feature of the system is its correction-driven continuous learning mechanism. Dental professionals can manually correct segmentation masks and detection bounding boxes through the graphical interface.

The system stores these corrections and triggers retraining when at least 20 corrected segmentation masks or 10 corrected detection annotations have accumulated. Difficult or uncertain samples receive priority during retraining.

The models are fine-tuned using transfer learning, with a learning rate of 10−410−4, a batch size of 4, and five training epochs. Early stopping and adaptive learning-rate scheduling help control overfitting and improve training stability.

The continuous learning workflow is:

Prediction → Clinician Correction → Data Storage → Selective Retraining → Model Update

5. Results and Performance

The YOLOv8 detection model achieved the following reported results:

These results indicate that the model can detect dental abnormalities with promising performance, although accuracy varies across categories. Cavity detection is particularly challenging because cavities represent a relatively small proportion of the annotated dataset.

6. Deployment

The system is deployed on Hugging Face Spaces using the Gradio framework. Its main technologies include:

• U-Net and TensorFlow: Tooth segmentation.

• YOLOv8 and Ultralytics: Dental pathology detection.

• Python: Backend processing and model integration.

• Gradio: Interactive web interface.

• Hugging Face Spaces: Cloud-based deployment.

Users can upload OPG images, view segmentation masks and detection bounding boxes with confidence scores, and provide corrections to support future model refinement. The system can use GPU acceleration when available and otherwise operate on a CPU.

Conclusion

This paper presents a dual-model deep learning framework integrating U-Net-based segmentation with YOLOv8-based detection for automated analysis of OPG radiographs. The segmentation model achieved 97.23% validation accuracy, while the detection model reached 78.7% mAP@0.5, demonstrating effective anatomical localization and multi-pathology detection. The combined framework provides complementary anatomical and pathological insights within a unified system. The inclusion of a correction-driven learning mechanism enables progressive model improvement through user feed-back, supporting practical clinical applicability. While the system shows promising performance for screening and documentation, challenges such as dataset imbalance, particularly for cavity detection remain. Fur-ther dataset expansion and clinical validation are required for reliable real-world deployment

References

[1] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, 2015, pp. 234–241. [2] G. Jocher, A. Chaurasia, and A. Qure, “YOLOv8: State-of-the-art real-time object detector,” Ultralytics, 2023. [Online]. Available: https://github.com/ultralytics/ultralytics [3] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [4] L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology, vol. 26, no. 3, pp. 297–302, 1945. [5] T.-Y. Lin et al., “Microsoft COCO: Common objects in con-text,” in European Conference on Computer Vision (ECCV), Springer, 2014, pp. 740–755. [6] J. Lee, H. Kim, and J. Park, “Deep learning in dental radio-graphy: Detection of dental caries using convolutional neural networks,” Journal of Dental Research, vol. 97, no. 11, pp. 1245–1252, 2018. [7] Y. Chen, L. Zhang, and Z. Li, “Automated tooth detection and numbering in panoramic radiographs using deep neural networks,” Computerized Medical Imaging and Graphics, vol. 80, 2020. [8] S. M. Khan, A. Rahman, and M. U. Akram, “Artificial in-telligence in dental radiology: A review of applications and challenges,” Dentomaxillofacial Radiology, vol. 49, no. 5, 2020. [9] Humans in the Loop, “Teeth Segmentation on Dental X-ray Images,” Kaggle, 2023. [Online]. Available: https://doi.org/10. 34740/KAGGLE/DSV/5884500 [10] IMT Kaggle Team, “Dental Radiography Dataset,” Kaggle, 2023. [Online]. Available: https://www.kaggle.com/datasets/ imtkaggleteam/dentalradiography/data. [Accessed: Apr. 2026].

Copyright

Copyright © 2026 Achyutha Nayaka D N, Ashray Chaitanya, Balaji M, Akshath Rai, Dr. Roopa H, Dr. Devi Ganesan. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.