Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
Similar Papers
Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)
Copied to clipboard
Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
| Challenge: | Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown. |
| Approach: | They propose an approach to edit and re-align alt-texts associated with images using human annotation. |
| Outcome: | The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks. |
What Makes for Good Image Captions? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a formal information-theoretic framework is developed for image captioning . the pyramid of captions is a method that generates enriched captions by integrating local and global visual information. |
| Approach: | They propose a formal information-theoretic framework for image captioning . they propose 'Pyramid of Captions' method that generates enriched captions . |
| Outcome: | The proposed framework provides a flexible foundation for analyzing and optimizing image captioning systems across diverse task requirements. |
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |
The Role of Data Curation in Image Captioning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing image captioning models treat all samples equally, neglecting mismatched data . Several other techniques have relied on curriculum learning strategies to adapt learning to the difficulty of the task. |
| Approach: | They propose to actively curate difficult samples in datasets using curriculum learning strategies to improve captioning models. |
| Outcome: | The proposed methods outperform existing models on the Flickr30K and COCO datasets. |
Denoising Large-Scale Image Captioning from Alt-text Data Using Content Selection Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent approaches to training large-scale image captioning (IC) models often fall short in terms of performance in leveraging noisy datasets in favor of clean annotations. |
| Approach: | They propose a technique that breaks down the task into two smaller, more controllable tasks - skeleton prediction and skelet-based caption generation. |
| Outcome: | The proposed method can generate better and denoised captions when using noisy datasets. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences (2025.acl-industry)
Copied to clipboard
Yusuke Hirota, Boyi Li, Ryo Hachiuma, Yueh-Hua Wu, Boris Ivanovic, Marco Pavone, Yejin Choi, Yu-Chiang Frank Wang, Yuta Nakashima, Chao-Han Huck Yang
| Challenge: | Large Vision-Language Models (LVLMs) have transformed image captioning . existing evaluations lack standardized criteria and a standardized evaluation framework . |
| Approach: | They propose a leaderboard for evaluating detailed captions that addresses three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. |
| Outcome: | The proposed model evaluates caption quality, descriptiveness, risks, and societal biases while tailoring criteria to user preferences. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |