Papers with Flickr30k
Learning Relation Alignment for Calibrated Cross-modal Retrieval (2021.acl-long)
Copied to clipboard
| Challenge: | despite advances in multimodal pre-training, cross-modal retrieval remains challenging . lack of relation consistency impairs contextualized representation of image-text pairs . |
| Approach: | They propose a new metric to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. |
| Outcome: | The proposed method boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin. |
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained models suffer from slow inference speed due to cross-modal attention in transformer architecture. |
| Approach: | They propose a multimodal approach that accelerates the inference time of ITR by thousands of times . they extract pre-cached feature indexes offline and employ instant dot-product matching online . |
| Outcome: | The proposed approach outperforms existing models that consume 1000 times magnitude of computational hours using the same features. |
Adaptive Weighted Proxy Tuning: Efficient Gray-Box Steering for Image Captioning. (2026.acl-industry)
Copied to clipboard
| Challenge: | Proxy tuning is a decoding-time approach that fails to account for instance-specific variations in model certainty and domain shift. |
| Approach: | They propose a gray-box steering framework that dynamically modulates the logit contributions of a large base model, a fine-tuned expert, and an untune . |
| Outcome: | Adaptive Weighted Proxy Tuning achieves performance parity with fine-tuned models while remaining parameter-free. |
An Empirical Study of Multimodal Model Merging (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that model merging can generate a multi-task solution without synchronous training. |
| Approach: | They propose to merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient architecture. |
| Outcome: | The proposed model merging outperforms naive models on various tasks with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% ADE20k. |
Image Embedding Sampling Method for Diverse Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, large-scale captioning models are less accessible for resource-constrained applications such as mobile devices and assistive technologies. |
| Approach: | They propose a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM as the backbone. |
| Outcome: | The proposed framework achieves comparable performance to larger models on MSCOCO, Flickr30k, and Nocaps test datasets while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions. |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour. |
| Approach: | They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions. |
| Outcome: | The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries. |