| Challenge: | Existing image captioning metrics focus on linguistic aspects and do not match human judgements at sentence-level. |
| Approach: | They propose to incorporate lexical and semantic metrics as features to capture adequacy and fluency of captions at different linguistic levels. |
| Outcome: | The proposed framework captures adequacy and fluency of captions at different linguistic levels. |
Similar Papers
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate image captions are English-centric, despite improvements in the CLIPScore metric . however, there are no available benchmarks for multilingual captioning evaluation . |
| Approach: | They propose to use machine-translated and machine-repurposed datasets to evaluate CLIPScore variants in multilingual settings. |
| Outcome: | The proposed evaluation strategies are based on machine-translated and human judgements. |
TIGEr: Text-to-Image Grounding for Image Caption Evaluation (D19-1)
Copied to clipboard
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
| Challenge: | Existing metrics based on text-level comparisons fail to assess the quality of captions produced by machines. |
| Approach: | They propose to use a machine-learned text-image grounding model to measure the accuracy of machine-generated captions and their correlation with human judgments. |
| Outcome: | The proposed metric has higher consistency with human judgments and is more accurate than existing metrics. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis (2021.acl-long)
Copied to clipboard
| Challenge: | Visual captioning is an open-ended area for evaluation, requiring specialized training to improve human-correlation. |
| Approach: | They propose a new evaluation framework rooted in information theory . they propose metric SPURTS and metric SMURF to measure fluency . |
| Outcome: | The proposed metrics achieve state-of-the-art correlation with human judgment compared with other evaluation metrics. |
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning approaches treat image-caption pairs indistinctly without considering the differences in their learning difficulties. |
| Approach: | They propose a pretrained vision–language model that measures cross-modal similarity and a model that uses cross-module similarity to measure the difficulty of captioning. |
| Outcome: | The proposed model achieves superior performance and competitive convergence speed to baselines without incurring additional training costs. |
EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified. |
| Approach: | They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness. |
| Outcome: | The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics. |
An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have proposed reference-free evaluations of image captions . however, these approaches are restrictive and favor captions with similar vocabulary but different meanings. |
| Approach: | They propose to use reference-free metrics to evaluate image captions . they propose to combine lexical overlap and semantics to identify fine-grained errors . |
| Outcome: | The proposed metrics struggle to identify fine-grained errors, the authors show . CLIPScore, UMIC, and PAC-S are sensitive to variations in image-relevant objects mentioned in the caption . |
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | N-gram-based evaluation metrics are unreliable due to low correlation to human judgments. |
| Approach: | They propose a metric that rewards correct details and penalizes incorrect ones. |
| Outcome: | The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient. |
What Makes for Good Image Captions? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a formal information-theoretic framework is developed for image captioning . the pyramid of captions is a method that generates enriched captions by integrating local and global visual information. |
| Approach: | They propose a formal information-theoretic framework for image captioning . they propose 'Pyramid of Captions' method that generates enriched captions . |
| Outcome: | The proposed framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. |
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer (D19-56)
Copied to clipboard
| Challenge: | Existing methods for textual transfer with no parallel corpora are insufficient to evaluate textual paraphrases with modified attributes or properties. |
| Approach: | They propose to add a metric for post-transfer classification accuracy and a method to combine them into a single overall score. |
| Outcome: | The proposed metrics correlate well with human judgments, at both the sentence-level and system-level. |