Image Caption Generator Using CNN and LSTM Hybrid Models

Abstract

Image captioning refers to the automated process of generating a descriptive sentence that conveys the content of a given image. The developed model receives an image as input and produces an English sentence that accurately represents what is depicted. This area has drawn considerable attention in recent years, particularly in the realm of cognitive computing, due to its reliance on both computer vision and natural language processing techniques. The system utilizes a Convolutional Neural Network (CNN) to analyze and extract visual features from the image, which are then passed to a Long Short- Term Memory (LSTM) network responsible for constructing the descriptive sentence. The CNN functions as the encoder, while the LSTM acts as the decoder. Following caption generation, the model\'s performance is evaluated to ensure the quality and relevance of the output. This enables the generation of meaningful, human-readable descriptions for various images.

Introduction

Image captioning is a task that combines computer vision and natural language processing to automatically generate descriptive sentences for images. The discussed system uses a deep learning approach, integrating:

• Convolutional Neural Networks (CNNs) for visual feature extraction.

• Long Short-Term Memory (LSTM) networks for generating captions in natural language.

The model is trained on the Flickr8k dataset, which contains images with multiple human-written captions. The system aims to generate grammatically correct, contextually relevant descriptions for each image.

2. Architecture

The model follows an encoder-decoder framework:

• Encoder (Image-Based Model):

  • Utilizes pre-trained CNNs like VGG16, Xception, or ResNet.

  • Extracts high-level features from the input image and outputs a feature vector.

• Decoder (Language-Based Model):

  • Uses an LSTM network to generate the caption sequentially.

  • Takes the feature vector and previously generated words to predict the next word.

  • Incorporates word embeddings for semantic understanding.

3. Key Components

• Image Preprocessing:

  • Resizes images to 224×224×3.

  • Applies normalization, grayscale conversion, noise reduction, thresholding, and edge detection.

• CNN Module:

  • Extracts key visual features (objects, actions, relationships).

  • Converts the image into a format compatible with the LSTM input.

• LSTM Module:

  • Generates captions word-by-word.

  • Trained using teacher forcing and optimized with cross-entropy loss.

  • During inference, uses greedy search or beam search to build complete captions.

• Caption Generation:

  • Produces final, coherent sentences from the outputs of the LSTM.

4. Enhancements and Techniques

• Attention Mechanism:

  • Allows the model to focus on specific parts of the image while generating each word.

  • Improves description accuracy, especially for complex scenes.

• Transfer Learning:

  • Leverages pre-trained CNNs for faster training and better performance.

5. Applications

• Assisting the visually impaired by describing images aloud.

• Enhancing image search engines.

• Automating content tagging and metadata generation.

6. Related Work

• Builds upon foundational models like Google’s Show and Tell (Inception + LSTM).

• Advances like Show, Attend and Tell introduced attention mechanisms for dynamic focus during caption generation.

• Multiple studies have validated the effectiveness of combining CNNs for image understanding with LSTMs for language modeling.

Conclusion

The Image Caption Generator project successfully showcases the combination of computer vision and natural language processing to produce meaningful textual descriptions of images. It utilizes a pre-trained VGG16 CNN model to extract rich visual features, which are then fed into an LSTM-based sequence model that constructs captions one word at a time.The application of transfer learning enhances the system\'s efficiency by reducing training duration and improving accuracy. Additionally, techniques such as tokenization and sequence padding ensure that the text data is structured appropriately for training the language model.Model performance is assessed through both qualitative analysis (by reviewing generated captions) and quantitative evaluation using BLEU scores, confirming that the outputs are contextually accurate and grammatically sound.This project emphasizes the effectiveness of deep learning in addressing complex, multimodal challenges and lays the groundwork for practical implementations like automated image tagging, assistive technologies for the visually impaired, and intelligent media organization tools.

References

[1] Base Paper: Katiyar, S., &Borgohain, S. K. (2021). Comparative evaluation of CNN architectures for image caption generation. arXiv preprint arXiv:2102.11506. [2] Kalena, P., Malde, N., Nair, A., Parkar, S., & Sharma, G. (2019). Visual Image Caption Generator Using Deep Learning. In 2nd International Conference on Advances in Science & Technology. [3] Xu, K., CA, U., Ba, J. L., CA, U., Kiros, R., EDU, T., &Bengio, Y. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Supplementary Material). [4] Jia, X., Gavves, E., Fernando, B., &Tuytelaars, T. (2015). Guiding the long-short term memory model for image caption generation. In Proceedings of the IEEE international conference on computer vision (pp. 2407-2415). [5] Yagcioglu, S., Erdem, E., Erdem, A., &Cak?c?, R. A Distributed Representation Based Query Expansion Approach for Image Captioning (Supplementary Material). [6] Kinghorn, P., Zhang, L., & Shao, L. (2018). A region-based image caption generator with refined descriptions. Neurocomputing, 272, 416-424.

Copyright

Copyright © 2025 Ganta Nikhila, Dr. M. Dhanalakshmi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.