VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words (2021.acl-long)
Copied to clipboard
| Challenge: | Existing text-to-image retrieval models lack good representations for textual and visual modalities. |
| Approach: | They propose a novel text-to-image retrieval model that uses a transformer to match images with textual queries. |
| Outcome: | The proposed model outperforms state-of-the-art models in MSCOCO and Flickr30K . it achieves substantial speed advantages for a 1 million image index . |
Similar Papers
VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Text-to-image retrieval is challenging because of cross-modal embeddings are bags of concepts, underrepresenting structured visual relationships. |
| Approach: | They propose a retrieval paradigm that embeds textual queries into the image modality via T2I generation and performs retrieval within the image mode to bypass weaknesses of cross-modal retrievers in recognizing subtle visual-spatial features. |
| Outcome: | The proposed retrieval paradigm outperforms previous approaches in visual-spatial retrieval benchmarks. |
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2022.tacl-1)
Copied to clipboard
| Challenge: | Current approaches to cross-modal retrieval process text and visual input jointly . current approaches are pretrained from scratch and suffer from huge retrieval latency and inefficiency issues . |
| Approach: | They propose a cooperative retrieve-and-rerank framework that turns pretrained text-image multi-modal models into efficient retrieval models. |
| Outcome: | The proposed framework improves retrieval performance over current approaches . it uses twin networks to encode all items of a corpus and a cross-encoder component for a more nuanced ranking . |
Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing text-image approaches use pre-trained vision-language representations for text retrieval . however, these models pose non-trivial memory requirements and substantial indexing time . |
| Approach: | They propose a framework to compress large pre-trained dual-encoders for lightweight text-image retrieval. |
| Outcome: | The proposed model performs better on Flickr30K and MSCOCO benchmarks than the original full model on mobile devices. |
Visual-Linguistic Dependency Encoding for Image-Text Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to image-text retrieval ignore semantic discrepancies caused by syntactic structure in natural language expressions and relationships among visual entities. |
| Approach: | They propose a visual-linguistic dependency encoder framework which explicitly models the dependency information among textual words and interaction patterns between image regions. |
| Outcome: | The proposed framework outperforms existing methods on a vision-linguistic compositional structure reasoning dataset. |
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour. |
| Approach: | They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions. |
| Outcome: | The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries. |
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal retrieval are mostly text-oriented, which lack the capability to process visual information. |
| Approach: | They propose a multi-modal multi-text embedding model VISTA which extends a powerful text encoder with the image understanding capability by introducing visual token embedds. |
| Outcome: | The proposed model achieves superior performance across a variety of multi-modal retrieval tasks in zero-shot and supervised settings. |
RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation (2025.findings-emnlp)
Copied to clipboard
Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, Nitzan Bitton Guetta, Almog Gueta, Royi Rassin, Dani Lischinski, Idan Szpektor
| Challenge: | Existing methods assess only one aspect of the task, misalign with human judgments or rely on costly API-based evaluation. |
| Approach: | RefVNLI evaluates textual alignment and subject preservation in a single run. |
| Outcome: | RefVNLI outperforms or matches existing baselines across multiple benchmarks and subject categories. |
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained models suffer from slow inference speed due to cross-modal attention in transformer architecture. |
| Approach: | They propose a multimodal approach that accelerates the inference time of ITR by thousands of times . they extract pre-cached feature indexes offline and employ instant dot-product matching online . |
| Outcome: | The proposed approach outperforms existing models that consume 1000 times magnitude of computational hours using the same features. |
Fusion of Detected Objects in Text for Visual Question Answering (D19-1)
Copied to clipboard
| Challenge: | Recent neural architectures such as Transformer and BERT allow for multimodal context, which may help model the meaning of words in general and also sharpen its understanding of instances of words. |
| Approach: | They propose a neural architecture that combines vision and natural language to advance models of multimodal context. |
| Outcome: | The proposed architecture achieves the highest performance on the Visual Commonsense Reasoning benchmark and the best performance to date on the public leaderboard. |
SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) relies on text-to-image retrieval using specialized bi-encoders . et al., 2022, 2024, 2021, 2023, 2026, 2030, 2040, 2050, 2060) document retrieval bridges human or artificial agents to the most relevant information, authors say . |
| Approach: | They propose a zero-shot visual document retrieval method that uses bi-encoders . they propose 63.4% nDCG@5 for visual document capture and a reusable semantic proxy . |
| Outcome: | The proposed method surpasses the strongest specialised multi-vector visual document encoder on the ViDoRe-v2 benchmark and scales similarly on MIRACL-VISION with broader multilingual coverage. |