Automated Image Caption Generator Using Deep Learning

Abstract

One of the most important tasks in computer vision and natural language processing is the automatic creation of image captions.. This paper presents an approach to automatically generate descriptive captions for images by combining Convolutional Neural Networks (CNNs) and Inception V3 architecture. The proposed system utilizes a pre-trained Inception V3 model to extract high-level features from input images. These extracted features are then passed to a Recurrent Neural Network (RNN), specifically an LSTM (Long Short-Term Memory) network, to generate coherent and contextually relevant captions. Inception V3, a deep convolutional neural network designed for large-scale image classification, serves as the feature extractor. It helps capture rich spatial hierarchies within the images, making it highly effective for understanding complex visual information. The LSTM network, on the other hand, is used to model the sequence of words in the caption, ensuring grammatical correctness and semantic accuracy. The system is trained on a large dataset of images paired with human-generated captions, such as the MS-COCO dataset, to ensure robust learning. The proposed method is evaluated based on its performance in generating captions that are semantically and syntactically appropriate. The model’s performance is compared to other existing image captioning methods, demonstrating its effectiveness in generating descriptive and accurate captions for unseen images. This work highlights the synergy between CNNs for visual feature extraction and LSTM networks for sequence generation, offering a promising solution for tasks requiring image-to-text conversion, including image retrieval, content-based indexing, and accessibility applications.

Introduction

The paper discusses the automatic image captioning task, which combines computer vision and natural language processing (NLP) to generate descriptive captions for images. Traditional methods relied on handcrafted features, but deep learning—especially Convolutional Neural Networks (CNNs) like InceptionV3—has significantly advanced feature extraction by capturing detailed image representations. However, CNNs alone cannot generate text sequences, so Long Short-Term Memory (LSTM) networks, a type of recurrent neural network (RNN), are employed to produce coherent, contextually relevant captions.

The research proposes an end-to-end image captioning system combining InceptionV3 for image feature extraction and LSTM for caption generation, trained on large datasets like MS-COCO. This approach enhances caption accuracy and relevance compared to previous models that often use ResNet. The system involves preprocessing images, extracting features, and generating captions via LSTM, with techniques like teacher forcing and beam search to improve training and inference.

The paper reviews related work, including evaluation metrics like BLEU, METEOR, CIDEr, and ROUGE-L, and highlights challenges such as dataset biases, context understanding, and computational demands. While existing models generate reasonable captions, limitations remain in fully grasping complex scenes and avoiding errors such as hallucination of non-existent objects.

Training results show that longer training (up to 50 epochs) improves caption relevance. The study suggests future improvements could come from attention mechanisms, transformer models, and reinforcement learning to further boost caption quality and fluency.

Conclusion

The automated image caption generator using deep learning effectively combines InceptionV3, a pretrained CNN, with an LSTM-based decoder to generate meaningful and contextually accurate captions. By leveraging the MS-COCO dataset, the system learns from diverse image-caption pairs, enabling it to describe new images with high relevance. Image processing techniques such as resizing, normalization, and feature extraction ensure that the model receives high-quality visual inputs, while beam search decoding enhances the fluency and coherence of generated captions. Despite its success, the model faces challenges in describing complex scenes and unseen objects due to its reliance on training data. Future improvements, including attention mechanisms, transformer-based architectures, and domain-specific fine- tuning, could further enhance its accuracy and applicability. Overall, this project highlights the potential of deep learning- based image captioning in improving image accessibility, AI-driven content generation, and assistive technologies.

References

[1] Sun Chengjian, Songhao Zhu, Zhe Shi, \"Image Annotation Via Deep Neural Network\", Published in: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), Pages 347- 350, DOI: 10.1109/MVA.2015.7153205. [2] Venkatesh N. Murthy, Subhransu Maji,R.Manmatha, \"Automatic Image Annotation using Deep Learning Representations\", Published in: ICMR \'15: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Pages 603-606, DOI: 10.1145/2671188.2749405 [3] Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan, \"Show and Tell: A Neural Image Caption Generator\", Published in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Pages 3156-3164,DOI: [4] Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio, \"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention\", Published in: Proceedings of the 32nd International Conference on Machine Learning (ICML), Volume 37, Pages 2048-2057,2015.URL:https://proceedings.mlr.press/v37/xuc15.ht ml. [5] Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu, \"BLEU: a Method for Automatic Evaluation of Machine Translation\", Published in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Pages 311-318, 2002. DOI: 10.3115/1073083.1073135. [6] Deep Learning Specialization, Offered by: deeplearning.ai, Available at: https://www.deeplearning.ai. [7] TensorFlow: An end-to-end open-source machine learning platform, Available at: https://www.tensorflow.org. [8] Gaurav, Pratistha Mathur, \"Empirical Study of Image Captioning Models Using Various Deep Learning Encoders\", Published in: Machine Learning and Computational Intelligence Techniques for Data Engineering, Lecture Notes in Electrical Engineering, Vol. 998, Pages 303-315, 2023 [9] Rashid Khan, Bingding Huang, Haseeb Hassan, Asim Zaman, Zhongfu Ye, \"A Comparative Study of Pre-trained CNNs and GRU-Based Attention for Image Caption Generation\", Published in: arXiv preprint arXiv:2310.07252, 2023. [10] Aditya Bhattacharya, Eshwar Shamanna Girishekar, Padmakar Anil Deshpande, \"Empirical Analysis of Image Caption Generation using Deep Learning\", Published in: arXiv preprint arXiv:2105.09906,2021. [11] Andrej Karpathy et al.(2017). Deep Visual-Semantic Alignments for Generating Image Descriptions\". Proceedings - 4th International Conference on Computing, Communication Control and descriptions(CVPR), 1–4. [12] Alec Radford et al. (2020)\"Learning Transferable Visual Models From Natural Language Supervision\" (CLIP): Advances, trends, applications, and datasets. The Visual Computer, 1–32.in axvix(2020) [13] Aditya Ramesh et al. (2022) \"Hierarchical Text-Conditional Image Generation with CLIP Latents“,arXiv Neural Processing Letters, 50(1), 103–119. Georgios Barlas, Christos Veinidis, and AviArampatzis. What we see in a photograph:content selection for image captioning. TheVisual Computer, 37(6):1309–1326, 2021. [14] Khaled Bayoudh, Raja Knani, Fay¸cal Ham-daoui, and Abdellatif Mtibaa. A surveyon deep multimodal learning for computervision: advances, trends, applications, anddatasets. The Visual Computer, pages 1–32,2021. [15] Rajarshi Biswas, Michael Barz, and DanielSonntag. Towards explanatory interac-tive image captioning using top-downand bottom-up features, beam search andre-ranking. KI-K¨unstliche Intelligenz,34(4):571–584, 2020.

Copyright

Copyright © 2025 G. Krishnaveni, Dr. G. Srinivasarao, K. Sita, R. Harika Priya, K. Usha Rani. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.