Communication breakdown: On the low mutual intelligibility between human and neural captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | 0-shot performance of a neural caption-based image retriever is higher when fed captions from a human-produced caption generator . despite the fact that the caption generator does not take the set of distractor images into account, this performance is only marginally above chance level. |
| Approach: | They compare the 0-shot performance of a neural caption-based image retriever with captions from a human-produced captioner. |
| Outcome: | The proposed model performs better when given human-produced captions or neural captions . the best pre-trained model perform better when fed captions produced by an out-of-the-box model . |
Similar Papers
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate image captions are English-centric, despite improvements in the CLIPScore metric . however, there are no available benchmarks for multilingual captioning evaluation . |
| Approach: | They propose to use machine-translated and machine-repurposed datasets to evaluate CLIPScore variants in multilingual settings. |
| Outcome: | The proposed evaluation strategies are based on machine-translated and human judgements. |
Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that account for perceptual differences in image captions are limited to use in English . culture-based tasks such as recognition, detection, and image retrieval are hindered by relying on English supervision. |
| Approach: | They propose and evaluate caption augmentation strategies to address these gaps . they use captions from german perception and captions that have been machine-translated or human-transcribed from English into german . |
| Outcome: | The proposed models achieve a mean recall improvement of +1.3, but still lack flexibility . cultural differences present in language with respect to object specificity and importance . |
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Visual language grounding is widely studied in modern neural image captioning systems . a novel algorithm for crafting adversarial examples in image captions is proposed . |
| Approach: | They propose an algorithm to craft adversarial examples in machine vision and perception . their approach provides two evaluation approaches to check if they can mislead systems . |
| Outcome: | The proposed algorithm can craft visually-similar adversarial examples with randomly targeted captions or keywords, and the results are transferable to other image captioning systems. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Language Resource Efficient Learning for Captioning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | XE loss and SC loss are both considered to be performance degradations for captioning tasks. |
| Approach: | They propose to generalize the single pairwise comparison in SC loss and use multiple generalized pairwise compares to reduce noise in baseline. |
| Outcome: | The proposed method outperforms state-of-the-art models on a video caption dataset using only half of the language resources. |
Non-neural Models Matter: a Re-evaluation of Neural Referring Expression Generation Systems (2022.acl-long)
Copied to clipboard
| Challenge: | In recent years, neural models have outperformed rule-based and classic approaches in NLG. |
| Approach: | They evaluate two English datasets and evaluate their performance using automatic and human evaluations. |
| Outcome: | The proposed model outperforms rule-based and classic approaches on two English datasets and is compared with human-based models. |
A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation (C18-1)
Copied to clipboard
| Challenge: | Recent studies have shown that multilingual NMT models can handle more than one translation direction with a single system. |
| Approach: | They propose a multilingual neural machine translation model that can handle more than one translation direction with a single system. |
| Outcome: | The proposed model performs well in low-resource settings against bilingual systems. |
What Makes for Good Image Captions? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a formal information-theoretic framework is developed for image captioning . the pyramid of captions is a method that generates enriched captions by integrating local and global visual information. |
| Approach: | They propose a formal information-theoretic framework for image captioning . they propose 'Pyramid of Captions' method that generates enriched captions . |
| Outcome: | The proposed framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. |
Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation (D18-1)
Copied to clipboard
| Challenge: | Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese–English news translation task. |
| Approach: | They empirically test neural machine translation on a Chinese–English news translation task . they show human raters prefer human over machine translation when evaluating documents . |
| Outcome: | The proposed method shows that human translators prefer document-level evaluation over machine translation . the results highlight the need to shift towards document- level evaluation as machine translation improves . |