simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to image captioning combine visual and semantic attention to generate a detailed and comprehensive caption. |
| Approach: | They propose a stepwise image-topic merging network that integrates visual and semantic attentions to generate a detailed caption. |
| Outcome: | The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performance. |
Similar Papers
AGIC: Attention-Guided Image Captioning to Improve Caption Relevance (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for image captioning generate generic captions that are limited in capturing nuanced visual details. |
| Approach: | They propose attention-guided image captioning which amplifies visual regions directly in the feature space to guide caption generation. |
| Outcome: | The proposed approach matches or surpasses state-of-the-art models while achieving faster inference. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
Transforming Visual Scene Graphs to Image Captions (2023.acl-long)
Copied to clipboard
Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang
| Challenge: | Existing approaches to generate captions using image captioning are based on multi-head attention (MHA) |
| Approach: | They propose to transform scene graphs into more descriptive captions by using multi-head attention to build a Graph Neural Network (GNN) . they construct a Mixture-of-Expert (MOE)-based decoder where each expert is built on MHA for discriminating the graph embeddings to generate different kinds of words. |
| Outcome: | The proposed framework can generate captions from multiple visual features and objects . it is based on a mixture-of-expert (MOE)-based decoder based upon MHA . |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Exploring the Impact of Vision Features in News Image Captioning (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent state-of-art models can achieve competitive performance even without vision features. |
| Approach: | They conduct extensive experiments with mainstream news image captioning models to determine whether vision features contribute to the generation of captions. |
| Outcome: | The proposed models can achieve competitive performance even without vision features. |
IDC: Boost Text-to-image Retrieval via Indirect and Direct Connections (2024.lrec-main)
Copied to clipboard
| Challenge: | Dual Encoders (DE) and Cross Attention (CA) frameworks for image and text retrieval are more accurate but slower. |
| Approach: | They propose a dual encoders-based approach to map image and text inputs into a coordinated representation space and calculate their similarity directly. |
| Outcome: | Extensive experiments on the MSCOCO and Flickr30K datasets validate the effectiveness of the proposed methods. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Improving Visual Question Answering by Referring to Generated Paragraph Captions (P19-1)
Copied to clipboard
| Challenge: | Empirical results show that paragraph captions help answer more visual questions . |
| Approach: | They propose a visual and textual question answering model which uses paragraph captions as input . they use cross-attention to extract related information, then consensus to fuse the inputs . |
| Outcome: | Empirical results show that paragraph captions help answer more visual questions . the proposed model significantly improves the baseline model . |
Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag (D19-1)
Copied to clipboard
| Challenge: | Existing models of video captioning use a network and semantics are mixed into one feature. |
| Approach: | They propose an Adaptive Semantic Guidance Network which instantiates whole video semantics to different POS-aware semantics with supervision of part of speech (POS) tag. |
| Outcome: | Extensive experiments show that the proposed model is more efficient than state-of-the-art models. |