Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Scientific feasibility assessment asks whether a claim aligns with established knowledge and whether experimental evidence could support or refute it. |
| Approach: | They frame scientific feasibility assessment as a diagnostic reasoning task . given a hypothesis, a model predicts feasible or infeasible and justifies its decision . they evaluate large language models under controlled knowledge conditions . |
| Outcome: | The results show that providing outcome evidence is more reliable than providing experiment descriptions. |
Similar Papers
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation approaches for large language models (LLMs) rely on existing tasks and benchmarks, raising concerns about test set contamination and the genuine comprehension abilities of LLMs. |
| Approach: | They propose to evaluate LLMs by designing new tasks, automatically generating evaluation datasets for the tasks, and conducting detailed error analyses to scrutinize LLM's adaptability to new tasks. |
| Outcome: | The proposed method examines LLMs’ adaptability to new tasks, their sensitivity to prompt variations, and their error tendencies. |
On Evaluating LLMs’ Capabilities as Functional Approximators: A Bayesian Evaluation Framework (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the way we can formulate tasks in text-in-text-out format. |
| Approach: | They propose a new evaluation framework to comprehensively assess LLMs’ function modeling abilities by adopting a Bayesian perspective of function modeling. |
| Outcome: | The proposed evaluation framework enables LLMs to excel in utilizing prior knowledge to develop a strong understanding of the underlying function. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |
Towards Reasoning in Large Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | Reasoning is a fundamental aspect of human intelligence that plays a crucial role in many intellectual activities. |
| Approach: | They propose to improve LLMs' ability to elicit reasoning by providing exemplars or prompts to model reasoning. |
| Outcome: | This paper provides a comprehensive overview of the state of knowledge on reasoning in large language models. |
Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains. |
| Approach: | They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies. |
| Outcome: | The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking. |
Evaluating the Deductive Competence of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models have limited abilities to solve deductive reasoning problems . performance differences between conditions do not improve overall performance . |
| Approach: | They investigate whether several large language models can solve a deductive reasoning problem in their conventional form. |
| Outcome: | The proposed models can solve a classic type of deductive reasoning problem in their conventional form. |
Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences (2024.lrec-main)
Copied to clipboard
| Challenge: | scholarly databases fail to aggregate, compare, contrast, and contextualize existing studies in service to a targeted research question. |
| Approach: | They propose to use large language models to discern evidence in support or refute of specific hypotheses based on abstracts. |
| Outcome: | The proposed method outperforms state-of-the-art methods and highlights opportunities for future research. |
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that large language models can be used to predict performance on new configurations. |
| Approach: | They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard . |
| Outcome: | The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set. |
Evaluating Large Language Models via Linguistic Profiling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) undergo extensive evaluation against various benchmarks collected in established leaderboards to assess their performance across multiple tasks. |
| Approach: | They propose a new evaluation methodology to test LLMs' sentence generation abilities under specific linguistic constraints. |
| Outcome: | The proposed evaluation methodology is based on the 'linguistic profiling' approach and is not intended to be a task-oriented evaluation. |