Papers with MS-COCO
Hashing based Efficient Inference for Image-Text Matching (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent work on image-text matching has focused on exploring interactions between images and sentences to improve performance without considering inference efficiency. |
| Approach: | They propose a hashing-based efficient inference module which can be plugged into existing frameworks to speed up inference step without reducing retrieval performance. |
| Outcome: | The proposed module can be plugged into existing framework to speed up inference step without reducing retrieval performance. |
Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective (2023.eacl-main)
Copied to clipboard
Zijian Zhang, Chang Shu, Ya Xiao, Yuan Shen, Di Zhu, Youxin Chen, Jing Xiao, Jey Han Lau, Qian Zhang, Zheng Lu
| Challenge: | Recent VSE models combine simple pooling methods with hard triplet loss to improve performance. |
| Approach: | They propose an adaptive pooling strategy that allows the model to learn how to aggregate features through a combination of simple pooling methods. |
| Outcome: | The proposed strategy outperforms current state-of-the-art systems on image-to-text and text-toimage retrieval. |
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search (2021.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on MS-COCO and Flickr30K benchmarks show that our methods significantly reduce the gender bias in image search models. |
| Approach: | They propose a fair sampling method and a feature clipping method to debias image search models. |
| Outcome: | The proposed methods significantly reduce gender bias in image search models on MS-COCO and Flickr30K benchmarks. |
Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to retrieve hard negative sentences are limited in the scale of the dataset thus fail to identify negative samples of high difficulty for every image. |
| Approach: | They propose to use a model to generate synthetic negative sentences with higher difficulty by masking and refilling the images and performing word discrimination and word correction tasks to improve retrieval and generation. |
| Outcome: | The proposed model generates synthetic negative sentences with higher difficulty on MS-COCO and Flickr30K and is robust and faithful to state-of-the-art training. |
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing image captioning datasets have limited cross-modal associations, preventing researchers from examining how inter-modal learning impacts intra-modal tasks. |
| Approach: | They propose to use image captioning data to support multi-modal retrieval training and evaluation to assess the impact of inter-modality learning. |
| Outcome: | The proposed model is able to measure the influence of intra- and inter-modality learning. |
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (C18-1)
Copied to clipboard
| Challenge: | Existing frameworks for grounding distributional representations of texts on the visual domain are limited . effective and efficient grounding of distributional embeddings remains challenging . |
| Approach: | They propose to ground distributional representations of texts on the visual domain using visual-semantic embeddings. |
| Outcome: | The proposed model improves on a diverse set of downstream tasks and defends known-type adversarial attacks. |
An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Position Embedding (PE) is essential for transformer to capture the sequence ordering of input tokens. |
| Approach: | They propose a unified position embedding method that bridges the semantic gap between modalities and embeds the anchor-based distance to guide computation of cross-attention. |
| Outcome: | The proposed method obtains new SOTA results on a wide range of benchmarks. |
Generative Text-to-Image Retrieval via Hierarchical Identifiers and Semantic Internalization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-image retrieval methods suffer from limited semantic discriminability, alignment bias, and closed-set restrictions. |
| Approach: | They propose a framework for semantic internalization for Generative Multimodal Alignment . they construct multi-granularity hierarchical identifiers to ensure unique, semantically consistent image representations . |
| Outcome: | The proposed framework outperforms state-of-the-art frameworks on Flickr30K and MS-COCO datasets . it achieves average Recall@1, Recall @5, and Recall_10 improvements of 10.65%, 8.50%, and 7.00% . |
Transforming Visual Scene Graphs to Image Captions (2023.acl-long)
Copied to clipboard
Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, Yu Zhang
| Challenge: | Existing approaches to generate captions using image captioning are based on multi-head attention (MHA) |
| Approach: | They propose to transform scene graphs into more descriptive captions by using multi-head attention to build a Graph Neural Network (GNN) . they construct a Mixture-of-Expert (MOE)-based decoder where each expert is built on MHA for discriminating the graph embeddings to generate different kinds of words. |
| Outcome: | The proposed framework can generate captions from multiple visual features and objects . it is based on a mixture-of-expert (MOE)-based decoder based upon MHA . |
Nearest Neighbor Normalization Improves Multimodal Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent training-free methods suggest that accuracy can be improved without fine-tuning. |
| Approach: | They propose a method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization. |
| Outcome: | The proposed method improves retrieval metrics for all contrastive models and datasets and does not require training on the reference database. |
Visual-Linguistic Dependency Encoding for Image-Text Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to image-text retrieval ignore semantic discrepancies caused by syntactic structure in natural language expressions and relationships among visual entities. |
| Approach: | They propose a visual-linguistic dependency encoder framework which explicitly models the dependency information among textual words and interaction patterns between image regions. |
| Outcome: | The proposed framework outperforms existing methods on a vision-linguistic compositional structure reasoning dataset. |
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-modal image-text retrieval struggle with nuanced cross-modal relationships. |
| Approach: | They propose a set-based approach that represents each sample with multiple embeddings to capture nuanced and diverse relationships. |
| Outcome: | The proposed method achieves state-of-the-art performance on MS-COCO and Flickr30k without external data. |
CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing mitigation approaches reduce hallucinated object mentions at the cost of degraded generation quality or require expensive retraining and task-specific supervision. |
| Approach: | They propose a lightweight framework for low-hallucination vision–language generation . it uses evidence-bounded minimal editing to revise or suppress unsupported referenced entities . |
| Outcome: | The proposed framework reduces hallucinations while maintaining or improving quality metrics. |