Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023.findings-acl)
Copied to clipboard
Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts
| Challenge: | Neural information retrieval (IR) systems have progressed rapidly in recent years . many IR benchmarks focus on downstream task accuracy, concealing costs incurred . |
| Approach: | They propose to include efficiency considerations on IR benchmarks to help drive progress . eral et al. propose to incorporate query latency and cost budgets into evaluation . |
| Outcome: | a new study shows that the best IR system varies according to how efficiency considerations are chosen and weighed . the proposed benchmarks would allow for more thorough exploration of possible system designs . |
Similar Papers
Neural Retriever and Go Beyond: A Thesis Proposal (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing neural retrievers are developed for pure-text queries, which prevents them from handling multi-modality queries. |
| Approach: | They propose methods to address issues of existing neural retrievers from three angles . they propose new model architectures, IR-oriented pretraining tasks and generating large scale training data . |
| Outcome: | The proposed methods address the abovementioned issues of neural retrievers from three angles and generate large scale training data. |
MAIR: A Massive Benchmark for Evaluating Instructed Retrieval (2024.emnlp-main)
Copied to clipboard
Weiwei Sun, Zhengliang Shi, Wu Long, Lingyong Yan, Xinyu Ma, Yiding Liu, Min Cao, Dawei Yin, Zhaochun Ren
| Challenge: | Existing IR benchmarks focus on a limited scope of tasks, making them insufficient for evaluating the latest IR models. |
| Approach: | They propose a multi-task instruction-tuned IR benchmark that includes 126 distinct IR tasks across 6 domains. |
| Outcome: | The proposed model performs better on instruction-tuned models than non-instruction-tunned models on MAIR. |
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Current information retrieval systems struggle to handle complex instructions, despite its critical importance . current models struggle to follow complex instructions in real-world applications, resulting in user-specific tasks. |
| Approach: | They propose a benchmark to evaluate instruction-following information retrieval in expert domains. |
| Outcome: | The proposed method improves on existing models and provides valuable insights to guide future advancements in retrieval. |
NevIR: Negation in Neural Information Retrieval (2024.eacl-long)
Copied to clipboard
| Challenge: | Negation is a common everyday phenomenon and has been a consistent area of weakness for language models. |
| Approach: | They ask IR models to rank two documents that differ only by negation . they find that most current information retrieval models do not consider negation. |
| Outcome: | The proposed benchmarks show that most current models do not consider negation . the results are similar to those found in the literature, but are poorer than random ranking . |
DAPR: A Benchmark on Document-Aware Passage Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Recent approaches to retrieve short passages are limited due to expensive operations such as selfattention. |
| Approach: | They propose to use a task to find passages from a large corpus of documents . they extend the SoTA passage retrievers with document context to achieve this task . |
| Outcome: | The proposed task is called Document-Aware Passage Retrieval (DAPR) it performs best on easy and hard queries, but fails on hard queries requiring document-context understanding . |
When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications (2023.acl-long)
Copied to clipboard
| Challenge: | Existing consensus on which OpenIE model is best for each application is lacking . different assumptions made by different models and datasets have a statistically significant effect on performance, making it important to choose the most appropriate OpenIE system for one’s applications. |
| Approach: | They propose to use OpenIE to extract relation tuples from plain text to compare different models and training sets to find the best model for their applications. |
| Outcome: | The proposed models perform well on a Complex QA application. |
Numbers Matter! Bringing Quantity-awareness to Retrieval Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantitative information is important for understanding documents and interpreting them. |
| Approach: | They propose two quantity-aware ranking techniques that rank both quantity and textual content . they use available retrieval systems to incorporate quantity information into queries . |
| Outcome: | The proposed methods can rank both quantity and textual content, either jointly or independently. |
Exploring the Practicality of Generative Retrieval on Dynamic Corpora (2024.emnlp-main)
Copied to clipboard
| Challenge: | a lack of comprehensive comparison between GR and Dual Encoders in IR systems is limiting . GR is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6) and storage footprint (x4) |
| Approach: | They propose to use autoregressive language models to perform information retrieval (IR) their results highlight the potential of GR for future use in practical IR systems . |
| Outcome: | The proposed model is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, efficient inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to the most common model, Dual Encoder (DE). |
Beyond [CLS] through Ranking by Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on generative ranking models for Information Retrieval has focused on discriminative methods that learn a similarity function to compare questions and candidates answers. |
| Approach: | They propose to use a language model to train a ranking function that model the semantic similarity of documents and queries instead of discriminative ranking functions. |
| Outcome: | The proposed approaches are as effective as state-of-the-art discriminative models for the answer selection task and show unlikelihood losses are reduced for IR. |
The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval (D19-1)
Copied to clipboard
| Challenge: | Existing studies do not investigate the effectiveness of MT metrics in predicting performance of downstream IR models. |
| Approach: | They examine the relationship between MT performance and IR quality in a CLIR-based system . they find that the choice of IR collection can significantly affect MT tuning decisions . |
| Outcome: | The proposed model can predict CLIR performance better from MT quality, the authors show . the proposed model is based on a BLEU-based model with a bag of words constraint . |