IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-only training methods overlook the modality gap between using text data during training and employing images during inference. |
| Approach: | They propose a novel approach that aligns text features with visually relevant features to mitigate the modality gap between using text data during training and employing images during inference. |
| Outcome: | The proposed method outperforms the state-of-the-art methods in image captioning and video captioning by a significant margin compared to training with text data. |
Similar Papers
Bridging Semantic and Modality Gaps in Zero-Shot Captioning via Retrieval from Synthetic Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot image captioning fail to bridge the semantic gap between images and textual inputs. |
| Approach: | They propose a retrieval-based framework that leverages only existing synthetic image-text pairs as its search corpus to bridge the gap when using synthetic data for captioning. |
| Outcome: | The proposed method bridges the semantic gap between a synthetic image and its input text . it extracts image-related textual descriptions to mitigate the modality gap during decoding . |
Text-Only Training for Image Captioning using Noise-Injected CLIP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to image captioning requires large datasets of captioned images and is difficult to collect. |
| Approach: | They propose to use a decoder to translate CLIP textual embeddings back into text . they show that this intuition is “almost correct” because of a gap between the embeddable spaces . |
| Outcome: | The proposed approach shows that the intuition is “almost correct” because of a gap between the embedding spaces, and rectifies this via noise injection during training. |
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
TROPE: TRaining-Free Object-Part Enhancement for Seamlessly Improving Fine-Grained Zero-Shot Image Captioning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to enhance zero-shot abilities in image captioning fail with fine-grained datasets. |
| Approach: | They propose a method to enhance captions with additional object-part details using object detector proposals and natural language processing techniques. |
| Outcome: | The proposed method improves performance on fine-grained datasets and improves on existing methods. |
SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) relies on text-to-image retrieval using specialized bi-encoders . et al., 2022, 2024, 2021, 2023, 2026, 2030, 2040, 2050, 2060) document retrieval bridges human or artificial agents to the most relevant information, authors say . |
| Approach: | They propose a zero-shot visual document retrieval method that uses bi-encoders . they propose 63.4% nDCG@5 for visual document capture and a reusable semantic proxy . |
| Outcome: | The proposed method surpasses the strongest specialised multi-vector visual document encoder on the ViDoRe-v2 benchmark and scales similarly on MIRACL-VISION with broader multilingual coverage. |
ELIOT: Zero-Shot Video-Text Retrieval through Relevance-Boosted Captioning and Structural Information Extraction (2025.naacl-srw)
Copied to clipboard
| Challenge: | Recent advances in video-text retrieval (VTR) have relied on supervised learning and fine-tuning. |
| Approach: | They propose a zero-shot video-text retrieval framework that leverages off-the-shelf captioners, large language models, and text retrieval methods without additional training or annotated data. |
| Outcome: | The proposed framework outperforms existing methods on video-text retrieval benchmarks without data. |
Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)
Copied to clipboard
Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
| Challenge: | Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown. |
| Approach: | They propose an approach to edit and re-align alt-texts associated with images using human annotation. |
| Outcome: | The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks. |
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023.findings-emnlp)
Copied to clipboard
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
| Challenge: | Existing methods for image-to-text generation store all knowledge within parameters, thus requiring computational-expensive fine-tuning. |
| Approach: | They propose a Retrieval-augmented Visual Language Model that stores all the knowledge within parameters and can be used to retrieve it from the external database. |
| Outcome: | The proposed model significantly boosts performance for image-to-text generation tasks with 4x less parameters compared with baseline methods. |
MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for supervised visual captioning require large scale of images or videos paired with descriptions in a specific language. |
| Approach: | They propose a zero-shot approach that generates captions for different scenarios without labeling . they use concept prompts to retrieve concepts and auto-encode them to learn writing styles . |
| Outcome: | The proposed approach generates captions for different scenarios and languages without labeled vision-caption pairs. |