Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows strong empirical advantages of dense retrieval in various information access scenarios, such as openQA. |
| Approach: | They propose a method which accumulates momentum negatives from past iterations and approximates future iteration with lookahead negatives as "teleportations" on web search and OpenQA, ANCE-Tele outperforms previous state-of-the-art systems of similar size and eliminates the dependency on sparse retrieval negatives. |
| Outcome: | The proposed method outperforms previous state-of-the-art systems on web search and OpenQA and is competitive among systems with significantly more parameters. |
Similar Papers
Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for building dense retrievers are often misaligned and do not reflect retrieval scenario at inference time. |
| Approach: | They propose a way to validate dense retrievers using a small subset of the entire corpus. |
| Outcome: | The proposed model improves top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval by 2 4 points for open-domain question answering. |
Negative Sampling Techniques in Dense Retrieval: A Survey (2026.findings-eacl)
Copied to clipboard
| Challenge: | Information Retrieval (IR) is fundamental to many modern NLP applications. |
| Approach: | They propose a taxonomy that categorizes negative sampling techniques in dense IR . they analyze them with respect to trade-offs between effectiveness, computational cost, implementation difficulty . |
| Outcome: | The proposed taxonomy categorizes techniques using random, static/dynamically mined, and synthetic datasets. |
Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on improving negative sampling strategy or extra pretraining for dense passage representations, but these studies are not capturing passage with internal representation conflicts. |
| Approach: | They propose a model with a smaller granularity to capture internal representation conflicts . they introduce a negative sampling strategy to encourage a diverse generation of sentence representations within the same passage. |
| Outcome: | The proposed model can be trained on three benchmark datasets to alleviate internal representation conflicts. |
Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural firststage retrieval models overcome lexical gap issue by projecting query and document to a shared dense space. |
| Approach: | They propose a multi-stage framework for neural passage retrieval using synthetic data, negative sampling, and fusion techniques. |
| Outcome: | The proposed framework improves retrieval accuracy and enhances the negative contrast in both stages. |
Query-as-context Pre-training for Dense Passage Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve passage retrieval performance by using context-supervised pre-training are weakly correlated. |
| Approach: | They propose to use query-as-context pre-training to train passage-query pairs . they evaluate the pre-trained models on large-scale passage retrieval benchmarks . |
| Outcome: | The proposed technique improves performance on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. |
Generative Dense Retrieval: Memory Can Be a Burden (2024.eacl-long)
Copied to clipboard
| Challenge: | Empirical results show that Generative Dense Retrieval (GDR) achieves an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability. |
| Approach: | They propose a Generative Dense Retrieval paradigm that auto-decodes document identifiers given a query and uses memory to avoid memory confusion. |
| Outcome: | Empirical results show that the proposed paradigm improves on the small-scale corpora and improves scalability. |
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2021.naacl-main)
Copied to clipboard
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, Haifeng Wang
| Challenge: | Open-domain question answering uses dense passage retrieval to find answers . however, it is difficult to effectively train a dual-encoder due to discrepancy between training and inference . |
| Approach: | They propose an optimized training approach to improve dense passage retrieval using RocketQA . they propose cross-batch negatives, denoised hard negatives and data augmentation . |
| Outcome: | The proposed approach outperforms state-of-the-art models on both MSMARCO and Natural Questions. |
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Dense retrievers have impressive performance, but their demand for abundant training data limits their application scenarios. |
| Approach: | They propose a method which uses unlabeled data to construct pseudo-positive examples from unlabelled data and then contrastively weighs the contrastive loss of different pairs according to the estimated relevance. |
| Outcome: | The proposed method beats the SOTA unsupervised Contriever model on BEIR and open-domain QA retrieval benchmarks and is a good few-shot learner. |
Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using LLMs to identify false negatives improves retrieval and reranker models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation. |
| Approach: | They use a simple, cost-effective approach to identify and relabel false negatives in training datasets. |
| Outcome: | The proposed approach improves retrieval models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation. |
Interpret and Control Dense Retrieval with Sparse Latent Features (2025.naacl-short)
Copied to clipboard
| Challenge: | Dense embeddings deliver strong retrieval performance but lack interpretability and controllability. |
| Approach: | They propose a novel approach using sparse autoencoders to interpret and control dense embeddings via latent sparsity. |
| Outcome: | The proposed approach retains the same retrieval accuracy as the original dense vectors, affirming their faithfulness. |