| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Similar Papers
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
ReCAP: Semantic Role Enhanced Caption Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Current vision language models lack specificity and overlook various aspects of the image. |
| Approach: | They propose to use semantic roles as control signals to guide captions to specific argument structures by focusing on specific objects and their associated semantic roles instead of general descriptions. |
| Outcome: | The proposed framework produces captions that exhibit enhanced quality, diversity, and controllability. |
AGIC: Attention-Guided Image Captioning to Improve Caption Relevance (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for image captioning generate generic captions that are limited in capturing nuanced visual details. |
| Approach: | They propose attention-guided image captioning which amplifies visual regions directly in the feature space to guide caption generation. |
| Outcome: | The proposed approach matches or surpasses state-of-the-art models while achieving faster inference. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
CapOnImage: Context-driven Dense-Captioning on Image (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning systems generate narrative captions for images, which are spatially detached from the image in presentation. |
| Approach: | They propose a task called captioning on image which generatesense captions at different locations of the image based on contextual information. |
| Outcome: | The proposed model achieves the best results in both captioning accuracy and diversity aspects. |
What Makes for Good Image Captions? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a formal information-theoretic framework is developed for image captioning . the pyramid of captions is a method that generates enriched captions by integrating local and global visual information. |
| Approach: | They propose a formal information-theoretic framework for image captioning . they propose 'Pyramid of Captions' method that generates enriched captions . |
| Outcome: | The proposed framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. |
Learning to Relate from Captions and Bounding Boxes (P19-1)
Copied to clipboard
| Challenge: | Existing methods for classifying images without supervision are limited. |
| Approach: | They propose a top-down attention mechanism to align entities in captions to objects in the image and leverage the syntactic structure of captions for alignment. |
| Outcome: | The proposed model achieves a recall@50 of 15% and recall@100 of 25% on the relationships present in the image and predicts relations that are not present in captions. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |