| Challenge: | Existing methods for evaluating image description generation systems are subjective and expensive to scale. |
| Approach: | They propose a new image-aware metric for evaluating image description generation systems . it estimates the faithfulness of a generated caption with respect to the content of the actual image . |
| Outcome: | The proposed metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on and rely exclusively on human references. |
Similar Papers
COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric. |
| Approach: | They propose a first discourse-aware learned generation metric for evaluating image descriptions. |
| Outcome: | The proposed metric predicts human ratings of captions on out-of-domain images. |
Transparent Human Evaluation for Image Captioning (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, Noah A. Smith
| Challenge: | Recent work has demonstrated that image captioning is a complex task that requires a large amount of human input. |
| Approach: | They develop a human evaluation protocol for image captioning models based on machine- and human-generated captions on the MSCOCO dataset. |
| Outcome: | The proposed model improves CLIPScore, a recent metric that uses image features, and improves human judgments because it is more sensitive to recall. |
Towards Automatic Evaluation for Image Transcreation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating image transcreation have relied on human evaluation. |
| Approach: | They propose a suite of automatic evaluation metrics inspired by machine translation metrics . they identify cultural relevance, semantic equivalence and visual similarity as critical dimensions of image transcreation . |
| Outcome: | The proposed evaluation metrics agree with human ratings across 7 countries. |
Learning-based Composite Metrics for Improved Caption Evaluation (P18-3)
Copied to clipboard
| Challenge: | Existing image captioning metrics focus on linguistic aspects and do not match human judgements at sentence-level. |
| Approach: | They propose to incorporate lexical and semantic metrics as features to capture adequacy and fluency of captions at different linguistic levels. |
| Outcome: | The proposed framework captures adequacy and fluency of captions at different linguistic levels. |
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020.acl-main)
Copied to clipboard
| Challenge: | Current approaches to interpretability evaluation focus on faithfulness criteria . current approaches focus on readability, plausibility and faithfulness . |
| Approach: | They argue that current binary definition of faithfulness sets unrealistic standards . they argue that a more graded definition would be of greater practical utility . |
| Outcome: | The proposed approach is based on three assumptions and lacks a graded definition of faithfulness. |
EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified. |
| Approach: | They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness. |
| Outcome: | The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics. |
TIGEr: Text-to-Image Grounding for Image Caption Evaluation (D19-1)
Copied to clipboard
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
| Challenge: | Existing metrics based on text-level comparisons fail to assess the quality of captions produced by machines. |
| Approach: | They propose to use a machine-learned text-image grounding model to measure the accuracy of machine-generated captions and their correlation with human judgments. |
| Outcome: | The proposed metric has higher consistency with human judgments and is more accurate than existing metrics. |
Improving Image Captioning Evaluation by Considering Inter References Variance (2020.acl-main)
Copied to clipboard
| Challenge: | Existing one-to-one metrics penalize mismatches without considering the intrinsic variance between ground truth captions. |
| Approach: | They propose a one-to-one metric based on BERTScore that could be extended to include new features for image captioning evaluation. |
| Outcome: | The proposed metric achieves state-of-the-art human judgment correlation while improving performance. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model (2024.acl-long)
Copied to clipboard
| Challenge: | Existing image captioning evaluation metrics do not provide an explanation for the assigned numerical score. |
| Approach: | They propose an explainable reference-free metric to provide an explanation for captions . they introduce score smoothing to align as closely as possible with human judgment . |
| Outcome: | The proposed metric achieves high correlations with human judgment across image captioning evaluation benchmarks and is publicly available at https://github.com/Yebin46/FLEUR. |