| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
Similar Papers
COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric. |
| Approach: | They propose a first discourse-aware learned generation metric for evaluating image descriptions. |
| Outcome: | The proposed metric predicts human ratings of captions on out-of-domain images. |
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
A Novel Computational Modeling Foundation for Automatic Coherence Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text coherence assessment rely on a proxy task . however, this approach does not capture the full range of factors contributing to coherency. |
| Approach: | They propose a formal linguistic definition of what makes a discourse coherent and formalize these conditions as respective computational tasks that are jointly trained. |
| Outcome: | The proposed model improves on two human-rated coherence benchmarks. |
How coherent are neural models of coherence? (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to model coherence are limited to small newswire corpora . evaluators need to be trained on lexical and document levels to perform evaluations . |
| Approach: | They propose four generic evaluation tasks that capture coherence-specific properties . they aim at capturing correct use of discourse connectives and lexical cohesion . |
| Outcome: | The proposed tasks capture coherence-specific properties, including correct use of discourse connectives, lexical cohesion, temporal consistency among events and participants in a story. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning approaches treat image-caption pairs indistinctly without considering the differences in their learning difficulties. |
| Approach: | They propose a pretrained vision–language model that measures cross-modal similarity and a model that uses cross-module similarity to measure the difficulty of captioning. |
| Outcome: | The proposed model achieves superior performance and competitive convergence speed to baselines without incurring additional training costs. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Visual Coherence Loss for Coherent and Visually Grounded Story Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing visual storytelling models fail to generate correct referring expressions for characters, causing 60% of the generated stories to be lacking local coherence. |
| Approach: | They propose a loss function inspired by a linguistic theory of coherence for self-supervised learning for image sequence representations and a feature matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. |
| Outcome: | The proposed features and loss function are effective for generating more coherent and visually grounded stories. |
A Cross-Domain Transferable Neural Coherence Model (P19-1)
Copied to clipboard
Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, Jackie Chi Kit Cheung
| Challenge: | Existing coherence models do not generalize to unseen categories of text . previous work advocates for generative models for cross-domain generalization . |
| Approach: | They propose a local discriminative neural model with a smaller negative sampling space that can discriminate against incorrect orderings. |
| Outcome: | The proposed model outperforms state-of-the-art methods on a standard benchmark dataset on the Wall Street Journal corpus and multiple challenging settings on Wikipedia articles. |