Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Dense passage retrieval improves ranking accuracy in open-domain question answering but at the cost of large space and memory requirements. |
| Approach: | They propose a simple unsupervised pipeline that includes principal component analysis (PCA), product quantization, and hybrid search to improve space efficiency. |
| Outcome: | The proposed pipeline achieves good accuracy–space trade-offs, for example, 48 compression with less than 3% drop in top-100 retrieval accuracy on average or 96 compression without drop in space requirements. |
Similar Papers
Dense Passage Retrieval for Open-Domain Question Answering (2020.emnlp-main)
Copied to clipboard
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih
| Challenge: | Open-domain question answering relies on efficient passage retrieval to select candidate contexts. |
| Approach: | They propose a dual-encoder framework that can be implemented to retrieve passages from a small number of questions and passages. |
| Outcome: | The proposed system outperforms a strong Lucene-BM25 system in top-20 passage retrieval accuracy on multiple open-domain QA benchmarks. |
CompAct: Compressing Retrieved Documents Actively for Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to condense extensive documents with no loss of information are difficult to implement in real-world scenarios. |
| Approach: | They propose a framework that employs an active strategy to condense extensive documents without losing key information. |
| Outcome: | The proposed framework improves performance and compression rate on multi-hop question-answering benchmarks. |
Extremely efficient online query encoding for dense retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing dense retrieval systems use the same model architecture for encoding both passages and queries, even though queries are much shorter and simpler than passages. |
| Approach: | They propose a small efficient RNN query encoder that can reduce latency by 12 with only a minor decrease in quality. |
| Outcome: | The proposed solution reduces latency by up to 12 while achieving 35.5 MRR@10 score. |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes (2021.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that dense representations outperform sparse representations with large index sizes. |
| Approach: | They propose to use dense low-dimensional representations to retrieve relevant documents . they show performance decreases quicker for increasing index sizes than for sparse representations . |
| Outcome: | The proposed representations outperform sparse representations with large index sizes. |
PISCO: Pretty Simple Compression for Retrieval-Augmented Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Document compression methods suffer from accuracy losses and limited context size. |
| Approach: | They propose a method that achieves a 16x compression rate with minimal accuracy loss . they show that PISCO outperforms existing compression models by 8% in accuracy . |
| Outcome: | The proposed method outperforms existing compression models by 8% in accuracy. |
Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods suffer from a lack of interpretability . we propose a new interpretability framework that decomposes dense embeddings into distinct, interpretable latent concepts. |
| Approach: | They propose a new interpretability framework that leveragesSparse Autoencoders to decompose uninterpretable dense embeddings fromDPR models into distinct, interpretable latent concepts. |
| Outcome: | The proposed interpretability framework achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches. |
Generative Dense Retrieval: Memory Can Be a Burden (2024.eacl-long)
Copied to clipboard
| Challenge: | Empirical results show that Generative Dense Retrieval (GDR) achieves an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability. |
| Approach: | They propose a Generative Dense Retrieval paradigm that auto-decodes document identifiers given a query and uses memory to avoid memory confusion. |
| Outcome: | Empirical results show that the proposed paradigm improves on the small-scale corpora and improves scalability. |
Efficient Passage Retrieval with Hashing for Open-domain Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Open-domain question answering systems often require large memory to run because of the massive size of their passage index. |
| Approach: | They propose a memory-efficient neural retrieval model that integrates a learning-to-hash technique into the state-of-the-art Dense Passage Retriever to represent the passage index using compact binary codes. |
| Outcome: | The proposed model significantly reduces memory cost from 65GB to 2GB without loss of accuracy on two open-domain question answering benchmarks. |
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-cuning techniques. |
| Approach: | They propose a pre-training architecture that learns to condense information into the dense vector through LM pre-training and a coCondenser architecture which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. |
| Outcome: | The proposed architecture reduces the need for heavy data engineering and large batch training. |