| Challenge: | Modern image captioning models are usually trained with text similarity objectives . reference captions often describe only the most salient objects in images . |
| Approach: | They propose to use CLIP to calculate multi-modal similarity and use it as a reward function . they propose a simple finetuning strategy to improve grammar that does not require extra text annotation. |
| Outcome: | The proposed model generates more distinctive captions than the CIDEroptimized model on text-to-image retrieval and fineCapEval. |
Similar Papers
Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to evaluate image captions are English-centric, despite improvements in the CLIPScore metric . however, there are no available benchmarks for multilingual captioning evaluation . |
| Approach: | They propose to use machine-translated and machine-repurposed datasets to evaluate CLIPScore variants in multilingual settings. |
| Outcome: | The proposed evaluation strategies are based on machine-translated and human judgements. |
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Contrastive language-image pre-training models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval. |
| Approach: | They propose a fine-tuning approach to enhance the representations of CLIP models for paraphrases by leveraging large language models. |
| Outcome: | The proposed model improves on baseline models across paraphrased retrieval, visual genome relation and attribution, and seven semantic textual similarity tasks. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio captioning lack fine-grained detail and contextual accuracy due to limited unimodal or superficial information. |
| Approach: | They propose a two-stage automated pipeline that uses pretrained models to extract contextual cues from video . a large language model synthesizes these inputs to generate detailed and context-aware captions . |
| Outcome: | The proposed method is scalable and generates detailed and context-aware captions on large-scale audio datasets. |
Pragmatic Inference with a CLIP Listener for Contrastive Captioning (2023.findings-acl)
Copied to clipboard
| Challenge: | a new method for contrastive captioning generates discriminative captions that distinguish target images from very similar alternative distractor images. |
| Approach: | They propose a pragmatic inference procedure that formulates captioning as a reference game between a speaker and a listener. |
| Outcome: | The proposed method outperforms previous methods for discriminative captioning by 11% to 15% accuracy in human evaluations. |
CLIP4IDC: CLIP for Image Difference Captioning (2022.aacl-short)
Copied to clipboard
| Challenge: | Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature extractor. |
| Approach: | They propose to transfer a CLIP model to the downstream IDC task to address two major issues: (1) a large domain gap exists between the pre-training datasets used for training such a visual feature extractor; (2) the visual feature extraction often does not effectively encode the visual changes between two images. |
| Outcome: | Experiments on three IDC benchmark datasets show the proposed model performs well. |
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | N-gram-based evaluation metrics are unreliable due to low correlation to human judgments. |
| Approach: | They propose a metric that rewards correct details and penalizes incorrect ones. |
| Outcome: | The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient. |
Focus! Relevant and Sufficient Context Selection for News Image Captioning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work only coarsely leverages the article to extract the necessary context, which makes it difficult for models to identify relevant events and named entities. |
| Approach: | They propose to use a vision and language retrieval model CLIP to localize the visually grounded entities in the news article and then capture the non-visual entities via an open relation extraction model. |
| Outcome: | The proposed model significantly improves on existing models and achieves state-of-the-art on multiple benchmarks. |
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Image captioning relies on reference-based automatic evaluations, but references are expensive to collect and comparing against multiple human-authored captions is insufficient. |
| Approach: | They propose a reference-free metric that can be used for automatic caption evaluation without references. |
| Outcome: | The proposed model outperforms existing metrics on image-text compatibility and a reference-augmented version achieves even higher correlation with human judgements. |
microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing UA methods for fine-grained image classification rely on coarse-grain visual tokens, which misses fine spatial details. |
| Approach: | They propose a label-free self-training framework that adapts visual features and LLMderived text prototypes using fine-grained cues. |
| Outcome: | The proposed framework improves alignment between finegrained visual regions and rich textual descriptions while updating only layer norms and a tiny head. |