| Challenge: | Existing studies on image and text retrieval using a dual-encoder model have not shown their effectiveness for fast inferences. |
| Approach: | They propose a dual-encoder model that connects vision and language in the same semantic space and integrates scene-text and visual information into a model. |
| Outcome: | The proposed model can interpret scene-text and surrounding visual information better than cross-encoder models. |
Similar Papers
Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing text-image approaches use pre-trained vision-language representations for text retrieval . however, these models pose non-trivial memory requirements and substantial indexing time . |
| Approach: | They propose a framework to compress large pre-trained dual-encoders for lightweight text-image retrieval. |
| Outcome: | The proposed model performs better on Flickr30K and MSCOCO benchmarks than the original full model on mobile devices. |
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour. |
| Approach: | They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions. |
| Outcome: | The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries. |
IDC: Boost Text-to-image Retrieval via Indirect and Direct Connections (2024.lrec-main)
Copied to clipboard
| Challenge: | Dual Encoders (DE) and Cross Attention (CA) frameworks for image and text retrieval are more accurate but slower. |
| Approach: | They propose a dual encoders-based approach to map image and text inputs into a coordinated representation space and calculate their similarity directly. |
| Outcome: | Extensive experiments on the MSCOCO and Flickr30K datasets validate the effectiveness of the proposed methods. |
Retrieval-Enhanced Dual Encoder Training for Product Matching (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent work has proposed a dual encoder for product matching due to its high performance and computation efficiency. |
| Approach: | They propose retrieval-enhanced dual encoder training to improve product matching . they use public and real-world product matching datasets to train the dual encoded model . |
| Outcome: | The proposed approach improves on a public and real-world product matching datasets. |
Sparse, Dense, and Attentional Representations for Text Retrieval (2021.tacl-1)
Copied to clipboard
| Challenge: | Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. |
| Approach: | They propose a dual-encoder-based neural model that combines the efficiency of dual encoders with expressiveness of more costly attentional architectures. |
| Outcome: | The proposed model outperforms strong alternatives in large-scale retrieval. |
GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to perform large-scale query-passage retrieval are term-based, but they lose interaction between query-pastage pairs. |
| Approach: | They propose to fuse query (passage) information into query representations via graph neural networks that are constructed by queries and their top retrieved passages. |
| Outcome: | The proposed model outperforms existing models on MSMARCO, Natural Questions and TriviaQA datasets and achieves the new state-of-the-art on these datasets. |
Multilingual Universal Sentence Encoder for Semantic Retrieval (2020.acl-demos)
Copied to clipboard
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, Ray Kurzweil
| Challenge: | Using a multi-task trained dual-encoder, our models embed text from 16 languages into a shared semantic space. |
| Approach: | They propose retrieval focused multilingual sentence embedding models on TensorFlow Hub. |
| Outcome: | The models achieve state-of-the-art on monolingual and cross-lingual retrieval (SR) and retrieval question answering (ReQA) competitive performance is obtained on related tasks of translation pair bitext retrieval and retrieving question answering. |
Pseudo-Relevance for Enhancing Document Representation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to document retrieval can be used to encode documents as vectors . a few query-relevant terms can be pruned out to reduce index overhead . |
| Approach: | They propose to enhance the document representation for the bi-encoder approach in dense document retrieval. |
| Outcome: | The proposed solution reduces latency and memory footprint up to 8- and 3-fold . it is validated on MSMARCO and real-world search query logs . |
Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale two-stream pre-trained models like CLIP have achieved tremendous success in image-text retrieval. |
| Approach: | They propose a cross-modal framework for image-text retrieval using two-stream pre-trained models . they embed images and texts into instance representations with two separate encoders . experimental results on MSCOCO and Flickr30k reveal the effectiveness of their framework . |
| Outcome: | The proposed framework improves image-text retrieval performance on two popular cross-modal retrieval benchmarks. |
Pushing the Limits of Radiology with Joint Modeling of Visual and Textual Information (P18-3)
Copied to clipboard
| Challenge: | Recent research has focused on the intersection of computer vision and natural language processing, but its adaption to the medical domain is not fully explored. |
| Approach: | They aim to develop machine learning models that can reason jointly on medical images and clinical text for advanced search, retrieval, annotation and description of medical images. |
| Outcome: | The proposed models can reason jointly on medical images and clinical text for advanced search, retrieval, annotation and description of medical images. |