FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model (2024.acl-long)
Copied to clipboard
| Challenge: | Existing image captioning evaluation metrics do not provide an explanation for the assigned numerical score. |
| Approach: | They propose an explainable reference-free metric to provide an explanation for captions . they introduce score smoothing to align as closely as possible with human judgment . |
| Outcome: | The proposed metric achieves high correlations with human judgment across image captioning evaluation benchmarks and is publicly available at https://github.com/Yebin46/FLEUR. |
Similar Papers
EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified. |
| Approach: | They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness. |
| Outcome: | The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics. |
VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for caption evaluation lack factual accuracy and limited context handling . VC-Inspector provides reproducible, fact-aware alternative that aligns closely with human judgments. |
| Approach: | They propose a lightweight, open-source large multimodal model for reference-free evaluation of video captions with a focus on factual accuracy. |
| Outcome: | Experiments show that VC-Inspector can generalize across diverse domains and improve on existing metrics. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have proposed reference-free evaluations of image captions . however, these approaches are restrictive and favor captions with similar vocabulary but different meanings. |
| Approach: | They propose to use reference-free metrics to evaluate image captions . they propose to combine lexical overlap and semantics to identify fine-grained errors . |
| Outcome: | The proposed metrics struggle to identify fine-grained errors, the authors show . CLIPScore, UMIC, and PAC-S are sensitive to variations in image-relevant objects mentioned in the caption . |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Learning-based Composite Metrics for Improved Caption Evaluation (P18-3)
Copied to clipboard
| Challenge: | Existing image captioning metrics focus on linguistic aspects and do not match human judgements at sentence-level. |
| Approach: | They propose to incorporate lexical and semantic metrics as features to capture adequacy and fluency of captions at different linguistic levels. |
| Outcome: | The proposed framework captures adequacy and fluency of captions at different linguistic levels. |
TIGEr: Text-to-Image Grounding for Image Caption Evaluation (D19-1)
Copied to clipboard
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
| Challenge: | Existing metrics based on text-level comparisons fail to assess the quality of captions produced by machines. |
| Approach: | They propose to use a machine-learned text-image grounding model to measure the accuracy of machine-generated captions and their correlation with human judgments. |
| Outcome: | The proposed metric has higher consistency with human judgments and is more accurate than existing metrics. |
InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing image captioning metrics provide a single score to measure caption qualities, which are less explainable and informative. |
| Approach: | They propose an Informative Metric for Reference-free Image Caption evaluation to support this feedback . they propose to provide a text precision score, a vision recall score and an overall quality score . |
| Outcome: | The proposed method improves on existing metrics on multiple benchmarks and compares coarse-grained scores with human judgements. |
VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for image captioning are primarily designed for short captions and are not suitable for long captions. |
| Approach: | They propose an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. |
| Outcome: | The proposed metric outperforms existing metrics and achieves superhuman performance on LongCap-Arena. |
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |