When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR (2025.findings-acl)
Copied to clipboard
Dayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim
| Challenge: | Dense retrievers encode text into embeddings to retrieve relevant documents . however, real-world corpora evolve, resulting in degraded retrieval performance . identifying when a dense retriever requires an update is critical for robust retrieval systems . |
| Approach: | They propose a task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. |
| Outcome: | The proposed method detects whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. |
Similar Papers
Exploring the Practicality of Generative Retrieval on Dynamic Corpora (2024.emnlp-main)
Copied to clipboard
| Challenge: | a lack of comprehensive comparison between GR and Dual Encoders in IR systems is limiting . GR is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6) and storage footprint (x4) |
| Approach: | They propose to use autoregressive language models to perform information retrieval (IR) their results highlight the potential of GR for future use in practical IR systems . |
| Outcome: | The proposed model is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, efficient inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to the most common model, Dual Encoder (DE). |
Reconstructing NER Corpora: a Case Study on Bulgarian (2020.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) and Named Enel Linking (NEL) are two related tasks that are under-resourced for the Slavic languages. |
| Approach: | They propose to use deep learning methods to improve a Named Entity Recognition corpus and to predict and annotate new types in a test corpus. |
| Outcome: | The proposed model improves a type-based Named Entity Recognition (NER) training corpus and predicts and annotates new types in a test corpus. |
Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for building dense retrievers are often misaligned and do not reflect retrieval scenario at inference time. |
| Approach: | They propose a way to validate dense retrievers using a small subset of the entire corpus. |
| Outcome: | The proposed model improves top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval by 2 4 points for open-domain question answering. |
How Good Are LLMs at Out-of-Distribution Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is crucial for ensuring AI safety . large language models (LLMs) are becoming more prevalent due to their scale, pre-training objectives, and paradigms used for inference. |
| Approach: | They propose to use large language models to investigate out-of-distribution (OOD) detection in machine learning. |
| Outcome: | The proposed method outperforms other OOD detectors in zero-grad and fine-tuning scenarios. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Generative Dense Retrieval: Memory Can Be a Burden (2024.eacl-long)
Copied to clipboard
| Challenge: | Empirical results show that Generative Dense Retrieval (GDR) achieves an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability. |
| Approach: | They propose a Generative Dense Retrieval paradigm that auto-decodes document identifiers given a query and uses memory to avoid memory confusion. |
| Outcome: | Empirical results show that the proposed paradigm improves on the small-scale corpora and improves scalability. |
Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks (2023.acl-srw)
Copied to clipboard
| Challenge: | Current approaches to OOD detection in NLP are not yet sufficiently sensitive to capture all samples characterized by various types of distributional shifts. |
| Approach: | They evaluated eight methods that are easily integrable into existing NLP systems and require no additional OOD data or model modifications. |
| Outcome: | The proposed methods are easily integrable into existing NLP systems and require no additional OOD data or model modifications. |
Types of Out-of-Distribution Texts and How to Detect Them (2021.emnlp-main)
Copied to clipboard
| Challenge: | Current NLP models produce unreliable or catastrophic predictions when training and test distributions differ . current models tend to produce unreliability or even catastrophic predictions that hurt user trust. |
| Approach: | They categorize examples as exhibiting a background shift or semantic shift and use calibration and density estimation methods to detect OOD examples. |
| Outcome: | The proposed methods beat calibration methods in background shift settings and perform worse in semantic shift settings. |
Alleviating Performance Degradation Caused by Out-of-Distribution Issues in Embedding-Based Retrieval (2025.findings-emnlp)
Copied to clipboard
Haotong Bao, Jianjin Zhang, Qi Chen, Weihao Han, Zhengxin Zeng, Ruiheng Chang, Mingzheng Li, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Recent studies reveal query out-of-distribution issues degrading ANN performance . a distribution regularizer is introduced into the encoder training objective to encourage alignment between query and base embeddings. |
| Approach: | They introduce a distribution regularizer into the encoder training objective to encourage alignment between query and base embeddings. |
| Outcome: | The proposed method consistently improves retrieval performance across multiple datasets. |
Rule Discovery for Natural Language Inference Data Generation Using Out-of-Distribution Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training rules for natural language inference do not cover the diversity of natural language. |
| Approach: | They propose a framework that combines out-of-distribution detection and clustering to identify new premise–hypothesis pairs in a dataset that are not covered by existing rules. |
| Outcome: | The proposed framework achieves +0.85%p accuracy on 2k and +0.15%p on 550k samples. |