| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Similar Papers
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning approaches treat image-caption pairs indistinctly without considering the differences in their learning difficulties. |
| Approach: | They propose a pretrained vision–language model that measures cross-modal similarity and a model that uses cross-module similarity to measure the difficulty of captioning. |
| Outcome: | The proposed model achieves superior performance and competitive convergence speed to baselines without incurring additional training costs. |
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
Learning to Relate from Captions and Bounding Boxes (P19-1)
Copied to clipboard
| Challenge: | Existing methods for classifying images without supervision are limited. |
| Approach: | They propose a top-down attention mechanism to align entities in captions to objects in the image and leverage the syntactic structure of captions for alignment. |
| Outcome: | The proposed model achieves a recall@50 of 15% and recall@100 of 25% on the relationships present in the image and predicts relations that are not present in captions. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
ReCAP: Semantic Role Enhanced Caption Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Current vision language models lack specificity and overlook various aspects of the image. |
| Approach: | They propose to use semantic roles as control signals to guide captions to specific argument structures by focusing on specific objects and their associated semantic roles instead of general descriptions. |
| Outcome: | The proposed framework produces captions that exhibit enhanced quality, diversity, and controllability. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)
Copied to clipboard
| Challenge: | Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language. |
| Approach: | They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization. |
| Outcome: | The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language. |