ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition (2026.findings-acl)
Copied to clipboard
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
| Challenge: | Large language models have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. |
| Approach: | They propose a benchmark for evaluating large language models on a sufficient set of scientific discovery sub-tasks. |
| Outcome: | The proposed framework extracts critical components from papers across 12 disciplines with expert validation confirming its accuracy. |
Similar Papers
A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing surveys on scientific LLMs focus on one or two fields or a single modality. |
| Approach: | They survey 260 scientific LLMs and examine their architectures and pre-training techniques . they also discuss commonalities and differences between LLM architectures . |
| Outcome: | The proposed model architectures and evaluation techniques are used to improve scientific discovery. |
Can Large Language Models Unlock Novel Scientific Research Ideas? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and ChatGPT have marked a turning point in the integration of Artificial Intelligence (AI) into people’s everyday lives. |
| Approach: | They conduct a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. |
| Outcome: | The proposed models generate more diverse ideas than GPT-4, GPT-3.5, and Gemini 1.0. |
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models (2026.acl-long)
Copied to clipboard
Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang
| Challenge: | Existing evaluations of large language models fail to reflect fine-grained capabilities . existing benchmarks are manually curated or domain-generic, limiting scalability and alignment with real use cases. |
| Approach: | They propose a framework that allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. |
| Outcome: | The proposed framework reveals fine-grained differences in scientific capabilities that standard benchmarks overlook . it allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific capabilities in LLMs. |
SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis (2025.findings-naacl)
Copied to clipboard
Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Wang Changxin, Zhifeng Gao, Hongshuai Wang, Li Yongge, Mujie Lin, Shuwen Yang, Jiankun Wang, Mingjun Xu, Jin Huang, Xi Fang, Jiaxi Zhuang, Yuqi Yin, Yaqi Li, Changhong Chen, Zheng Cheng, Zifeng Zhao, Linfeng Zhang, Guolin Ke
| Challenge: | Existing benchmarks fail to adequately evaluate the proficiency of Large Language Models (LLMs) Existing standards do not cover the skills needed to evaluate LLMs in scientific literature analysis. |
| Approach: | They propose a benchmark to evaluate the proficiency of large language models in scientific literature analysis. |
| Outcome: | SciAssess evaluates 11 LLMs on multiple tasks across scientific fields. |
Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences (2024.lrec-main)
Copied to clipboard
| Challenge: | scholarly databases fail to aggregate, compare, contrast, and contextualize existing studies in service to a targeted research question. |
| Approach: | They propose to use large language models to discern evidence in support or refute of specific hypotheses based on abstracts. |
| Outcome: | The proposed method outperforms state-of-the-art methods and highlights opportunities for future research. |
Large Language Models for Generative Recommendation: A Survey and Visionary Discussions (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing but are not fully able to leverage the generative power of LLM. |
| Approach: | They examine the progress, methods, and future directions of large language models . they examine what generative recommendation is, why RS should advance to generative recommendations . |
| Outcome: | The proposed approach can be simplified to generate recommendations from the entire pool of items. |
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs (2025.acl-long)
Copied to clipboard
Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. |
| Approach: | They propose a Chinese Comprehensive Constraints Following Benchmark for LLMs that compiles constraints from real-world instructions and constructs a systematic framework for constraint types. |
| Outcome: | The proposed framework integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. |
Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on performance benchmarks without fully comparing LLMs to graph learning models. |
| Approach: | They evaluate off-the-shelf and instruction-tuned graph learning models across a variety of scenarios. |
| Outcome: | The proposed models outperform traditional graph learning models in few-shot settings, the authors show . their models out perform models with instruction tuning, and they show excellent generalization and robustness. |
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics (2026.acl-long)
Copied to clipboard
Yuxing Lu, Xukai Zhao, J. Ben Tamo, Micky C. Nnamdi, Rui Peng, Shuang Zeng, Xingyu Hu, Jinzhuo Wang, May Dongmei Wang
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities on general text, but their proficiency in specialized scientific domains remains uncharacterized. |
| Approach: | They evaluate the capabilities of large language models in metabolomics research using MetaBench . they found that models perform well on text generation tasks, but cross-database identifier grounding remains challenging . |
| Outcome: | The evaluation of 25 open- and closed-source LLMs reveals distinct performance patterns across metabolomics tasks. |