| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
Similar Papers
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Neural Caption Generation for News Images (L18-1)
Copied to clipboard
| Challenge: | Existing methods for automatic caption generation of images are lacking in the field of image-related applications. |
| Approach: | They propose a method for automatically generating captions for news images . they propose several deep neural network architectures built upon Recurrent Neural Networks . |
| Outcome: | The proposed method outperforms a traditional method on a BBC News dataset using automatic evaluation and human evaluation. |
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Journalistic Guidelines Aware News Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that JoGANIC outperforms state-of-the-art methods for image caption generation. |
| Approach: | They propose a method to generate descriptive and informative captions for news article images . they leverage the structure of captions to improve the generation quality and guide their representation . |
| Outcome: | The proposed method outperforms state-of-the-art methods on two large-scale datasets. |
Image Caption Generation for News Articles (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on news-image captioning requires a joint understanding of image and text. |
| Approach: | They propose a Transformer model that integrates text and image modalities and attends to textual features from visual features in generating a caption. |
| Outcome: | The proposed model outperforms the state-of-the-art model and improves the quality of news-image captions. |
CapOnImage: Context-driven Dense-Captioning on Image (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning systems generate narrative captions for images, which are spatially detached from the image in presentation. |
| Approach: | They propose a task called captioning on image which generatesense captions at different locations of the image based on contextual information. |
| Outcome: | The proposed model achieves the best results in both captioning accuracy and diversity aspects. |
Focus! Relevant and Sufficient Context Selection for News Image Captioning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work only coarsely leverages the article to extract the necessary context, which makes it difficult for models to identify relevant events and named entities. |
| Approach: | They propose to use a vision and language retrieval model CLIP to localize the visually grounded entities in the news article and then capture the non-visual entities via an open relation extraction model. |
| Outcome: | The proposed model significantly improves on existing models and achieves state-of-the-art on multiple benchmarks. |
Exploring the Impact of Vision Features in News Image Captioning (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent state-of-art models can achieve competitive performance even without vision features. |
| Approach: | They conduct extensive experiments with mainstream news image captioning models to determine whether vision features contribute to the generation of captions. |
| Outcome: | The proposed models can achieve competitive performance even without vision features. |