Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a small number of gallery data points are frequently retrieved, resulting in a decline in retrieval performance. |
| Approach: | They propose a framework that leverages both gallery and query data to address hubness . they propose dual inverted softmax and dual dynamic inverted hardmax methods to normalize similarity . |
| Outcome: | The proposed framework reduces the occurrence of hubs during inference while improving similarity between non-hubs and queries. |
Similar Papers
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2022.tacl-1)
Copied to clipboard
| Challenge: | Current approaches to cross-modal retrieval process text and visual input jointly . current approaches are pretrained from scratch and suffer from huge retrieval latency and inefficiency issues . |
| Approach: | They propose a cooperative retrieve-and-rerank framework that turns pretrained text-image multi-modal models into efficient retrieval models. |
| Outcome: | The proposed framework improves retrieval performance over current approaches . it uses twin networks to encode all items of a corpus and a cross-encoder component for a more nuanced ranking . |
Normalized Contrastive Learning for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. |
| Approach: | They propose a normalized contrastive learning algorithm that normalizes the sum retrieval probabilities of each instance so that every text and video instance is fairly represented. |
| Outcome: | Empirical results show that NCL brings significant gains in text-video retrieval on different model architectures without any architecture engineering. |
LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal retrieval is essential for interpreting cultural heritage data, but its effectiveness is limited by incomplete or inconsistent textual descriptions. |
| Approach: | They propose a data augmentation framework that enhances cross-modal retrieval performance by improving the completeness and consistency of LLM-generated descriptions. |
| Outcome: | The proposed framework improves cross-modal retrieval performance by improving completeness and consistency of LLM-generated descriptions. |
InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that multi-modal data representations tend to cluster within a limited convex cone, hindering retrieval performance. |
| Approach: | They propose a method that uses graph convolution and average pooling to separate representations by increasing distances between data points. |
| Outcome: | The proposed method reduces representation degeneration problem by increasing distances between data points. |
One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness (2026.acl-long)
Copied to clipboard
| Challenge: | et al., 2010) show that hub embeddings are close to many unrelated examples in high-dimensional embeddable spaces . cross-modal encoders that project different modalities into a shared space are useful for cross-module applications . |
| Approach: | They propose a method for identifying the hub embedding and its corresponding hub text . they use images to evaluate cross-modal encoders that project different modalities into a shared space . |
| Outcome: | The proposed method can identify a single hub embedding and its corresponding hub text . it achieves comparable or higher similarity scores than human-written reference captions in many images . |
DEBAR: Mitigating Contextual Bias in Cross-Document Relation Extraction via Dual-Stream Decoupling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods focus on sentence-level or singledocument settings, resulting in one-sided relation transfer contextual bias and incomplete reasoning chains. |
| Approach: | They propose a framework to explicitly decouple and preserve bidirectional bridge evidence and a dynamic loss optimization objective to separate head and tail contexts. |
| Outcome: | The proposed framework decouples and preserves bidirectional bridge evidence while capturing global dependencies through iterative message passing. |
CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval (2026.findings-acl)
Copied to clipboard
Zhipeng Qian, Zihan Liang, Yufei Ma, Ben Chen, Huangyu Dai, Yiwei Ma, Jiayi Ji, Chenyi Lei, Han Li, Xiaoshuai Sun
| Challenge: | Existing approaches to search for images using single-modality are limited by representation space fragmentation. |
| Approach: | They propose a unified representation framework that achieves efficient query-target alignment . they introduce a multi-level Chain-of-Thought prompting strategy that guides MLMs to generate discriminative, semantically compatible captions for target images . |
| Outcome: | The proposed framework achieves efficient query-target alignment through synergistic components. |
Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-modal image-text retrieval struggle with nuanced cross-modal relationships. |
| Approach: | They propose a set-based approach that represents each sample with multiple embeddings to capture nuanced and diverse relationships. |
| Outcome: | The proposed method achieves state-of-the-art performance on MS-COCO and Flickr30k without external data. |
Cross-Lingual Cross-Modal Consolidation for Effective Multilingual Video Corpus Moment Retrieval (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing multilingual video corpus moment retrieval methods are based on a two-stream structure. |
| Approach: | They propose a multilingual video corpus moment retrieval task that uses a two-stream structure to generate a query-visual similarity and a subtitle stream exploits the query-subtitle similarity. |
| Outcome: | The proposed method improves accuracy on a large-scale video corpus moment retrieval dataset. |
End-to-end Knowledge Retrieval with Multi-modal Queries (2023.acl-long)
Copied to clipboard
| Challenge: | a new task is proposed to learn knowledge retrieval with multimodal queries . a vision-language model can retrieve knowledge using images and text inputs . |
| Approach: | They propose a task for vision-language models to retrieve knowledge with multi-modal queries . they propose reViz, a model that integrates content from both text and image queries based on a multimodal query task . |
| Outcome: | The proposed task performs better under zero-shot settings than previous work on cross-modal retrieval. |