PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing image captioning metrics are vulnerable to lexical perturbations, but they are not robust to such perturbations. |
| Approach: | They propose a perturbation-robust multilingual CLIPScore which is a reference-free image captioning metric for multiple languages. |
| Outcome: | The proposed metric outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages while maintaining a strong correlation with human judgments. |
Similar Papers
An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have proposed reference-free evaluations of image captions . however, these approaches are restrictive and favor captions with similar vocabulary but different meanings. |
| Approach: | They propose to use reference-free metrics to evaluate image captions . they propose to combine lexical overlap and semantics to identify fine-grained errors . |
| Outcome: | The proposed metrics struggle to identify fine-grained errors, the authors show . CLIPScore, UMIC, and PAC-S are sensitive to variations in image-relevant objects mentioned in the caption . |
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | N-gram-based evaluation metrics are unreliable due to low correlation to human judgments. |
| Approach: | They propose a metric that rewards correct details and penalizes incorrect ones. |
| Outcome: | The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient. |
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate image captions are English-centric, despite improvements in the CLIPScore metric . however, there are no available benchmarks for multilingual captioning evaluation . |
| Approach: | They propose to use machine-translated and machine-repurposed datasets to evaluate CLIPScore variants in multilingual settings. |
| Outcome: | The proposed evaluation strategies are based on machine-translated and human judgements. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
Learning-based Composite Metrics for Improved Caption Evaluation (P18-3)
Copied to clipboard
| Challenge: | Existing image captioning metrics focus on linguistic aspects and do not match human judgements at sentence-level. |
| Approach: | They propose to incorporate lexical and semantic metrics as features to capture adequacy and fluency of captions at different linguistic levels. |
| Outcome: | The proposed framework captures adequacy and fluency of captions at different linguistic levels. |
Do Image–Text Metrics Respect Semantic Invariances? (2026.findings-acl)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu, Jyotika Singh, Karan Dua, Hansa Meghwani, Matthew Rowe, M. Avendi, Yassi Abbasi, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Reference-free image–to–text evaluators are now standard for scoring image–caption alignment, yet it is unclear whether they respect semantic invariances. |
| Approach: | They propose an invariance probe on five popular evaluators under semantics-preserving perturbations along three axes: spatial edits, object changes, and socio-linguistic framing. |
| Outcome: | The proposed invariance probe shows that spatial edits and simple phrasing changes shift scores by ()6% on average and cause ranking flips in up to (),37% of cases. |
A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing caption evaluation metrics lack granular assessments for errors within captions . lack of uncertainty quantification can give way to misleading scores, reducing user trust . |
| Approach: | They propose a conformal risk control framework to generate and calibrate CLIPScore distributions . they use a model-agnostic conformal framework to detect erroneous words . |
| Outcome: | The proposed method detects erroneous words while providing formal guarantees aligned with desired risk levels. |
Evaluating Robustness to Input Perturbations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that Neural Machine Translation models are brittle to small perturbations in the input. |
| Approach: | They propose to use subword regularization to measure the relative degradation and changes in translation when perturbations are added to the input. |
| Outcome: | The proposed measures show that the models are more robust to perturbations when subword regularization methods are used. |
Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in multimodal systems have demonstrated remarkable capabilities in generating multimodal content from multimodal inputs. |
| Approach: | They propose a benchmark that leverages large language models to generate deceptive text samples to exploit compositional vulnerabilities across different modalities. |
| Outcome: | The proposed approach exploits compositional vulnerabilities across images, videos, and audios. |