LitSearch: A Retrieval Benchmark for Scientific Literature Search (2024.emnlp-main)
Copied to clipboard
| Challenge: | Literature search questions pose significant challenges for modern retrieval systems . a lack of domain expertise and reasoning through lengthy papers is a challenge . |
| Approach: | They propose a retrieval benchmark for literature search queries using inline citations from papers and questions about recently published papers. |
| Outcome: | The proposed retrieval benchmarks outperform state-of-the-art retrieval models and reranking pipelines. |
Similar Papers
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods is presented. |
| Approach: | They evaluate 22 reranking methods including 40 variants across established benchmarks . primary goal is to determine whether performance disparity exists between LLM-based reranters and lightweight counterparts based on novel queries . |
| Outcome: | The proposed methods perform better on familiar queries than lightweight models, the authors show . |
RELiC: Retrieving Evidence for Literary Claims (2022.acl-long)
Copied to clipboard
| Challenge: | Literature scholars often provide evidence for claims they make in the form of quotations from a work . a novel evidence retrieval task requires a deep understanding of complex literary phenomena . |
| Approach: | They use a large-scale dataset to formulate a task of literary evidence retrieval . they implement a RoBERTa-based dense passage retriever that outperforms existing methods . |
| Outcome: | The proposed model outperforms pretrained information retrieval baselines, but there is room for improvement. |
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |
Unsupervised Dense Retrieval for Scientific Articles (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing lexical search models suffer from lexica gap problems and are not fast enough to solve these problems. |
| Approach: | They build a dense retrieval based semantic search engine on scientific articles from Elsevier that generates high-quality pseudo training labels. |
| Outcome: | The proposed model significantly outperforms the currently deployed lexical search engine on the two test sets. |
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition (2026.findings-acl)
Copied to clipboard
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
| Challenge: | Large language models have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. |
| Approach: | They propose a benchmark for evaluating large language models on a sufficient set of scientific discovery sub-tasks. |
| Outcome: | The proposed framework extracts critical components from papers across 12 disciplines with expert validation confirming its accuracy. |
SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis (2025.findings-naacl)
Copied to clipboard
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Wang Changxin, Zhifeng Gao, Hongshuai Wang, Li Yongge, Mujie Lin, Shuwen Yang, Jiankun Wang, Mingjun Xu, Jin Huang, Xi Fang, Jiaxi Zhuang, Yuqi Yin, Yaqi Li, Changhong Chen, Zheng Cheng, Zifeng Zhao, Linfeng Zhang, Guolin Ke
| Challenge: | Existing benchmarks fail to adequately evaluate the proficiency of Large Language Models (LLMs) Existing standards do not cover the skills needed to evaluate LLMs in scientific literature analysis. |
| Approach: | They propose a benchmark to evaluate the proficiency of large language models in scientific literature analysis. |
| Outcome: | SciAssess evaluates 11 LLMs on multiple tasks across scientific fields. |
Scientific Paper Retrieval with LLM-Guided Semantic-Based Ranking (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies also use large language models (LLMs) for query understanding, but these methods lack grounding in corpus-specific knowledge and may generate unreliable or unfaithful content. |
| Approach: | They propose a paper retrieval framework that combines large language models (LLMs) with a concept-based semantic index to capture scientific concepts. |
| Outcome: | The proposed framework improves the performance of various base retrievers, surpasses strong existing LLM-based baselines, and remains highly efficient. |
Open-World Evaluation for Retrieving Diverse Perspectives (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing retrieval systems only cover diverse perspectives on 33.74% of the examples . existing systems only focus on relevance to the question, ignoring diversity. |
| Approach: | They build a Benchmark for Retrieval Diversity for Subjective questions (BERDS) based on a question and diverse perspectives associated with the question . they evaluate retrievers paired with a corpus to determine whether each document contains a perspective . |
| Outcome: | The proposed approach improves retrieval diversity on complex questions . existing retrieval systems only cover diverse perspectives on 33.74% of the examples . |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |
MIR: Methodology Inspiration Retrieval for Scientific Research Problems (2025.acl-long)
Copied to clipboard
Aniketh Garikaparthi, Manasi Patwardhan, Aditya Sanjiv Kanade, Aman Hassan, Lovekesh Vig, Arman Cohan
| Challenge: | Existing methods for generating ideas rely on grounding the discovery process within the literature, but their effectiveness varies significantly with the quality and nature of the retrieved literature. |
| Approach: | They construct a methodological inspiration retrieval task using a citation-based methodology adjacency graph and embed an "intuitive prior'' into dense retrievers. |
| Outcome: | The proposed method achieves significant gains in Recall@3 and mAP over strong baselines. |