Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)
Copied to clipboard
| Challenge: | a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision. |
| Approach: | They introduce Physics, a benchmark for university-level physics problem solving. |
| Outcome: | The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems . |
Similar Papers
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning (2025.acl-long)
Copied to clipboard
Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, Jun Liu
| Challenge: | Large language models demonstrate remarkable capabilities across various domains, including mathematics and logic reasoning. |
| Approach: | They propose a physics-based reasoning benchmark that includes physics theorems and constraints and a Physics Solution Auto Scoring Framework to evaluate physics based reasoning in large language models. |
| Outcome: | The proposed framework enables models to achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.99%). |
Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the potential of retrieval-augmented generation (RAG) with foundation models to enhance expert-level reasoning. |
| Approach: | They introduce PhoPile, a high-quality multimodal dataset specifically designed for Olympiad-level physics. |
| Outcome: | The proposed model can be used to solve Olympiad-level physics problems. |
FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for numerical reasoning often lack explicit knowledge of formulas . current datasets do not provide process supervision information, resulting in incomplete reasoning . |
| Approach: | They propose a benchmark for formula-based numerical reasoning with 5,324 questions . they provide annotations in English and Chinese and a formula database as an external knowledge source . |
| Outcome: | The proposed model includes 5,324 questions requiring calculations grounded in external physics principles. |
UTMath: A Benchmark for Math Evaluation with Unit Test (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prevailing benchmarks for mathematical reasoning include MATH and AIME . predicated on single-instantiation problems with fixed numbers, these models leave generalization on isomorphic problem variants untested. |
| Approach: | They propose a mathematical reasoning benchmark that quantifies solution accuracy and solution space generality. |
| Outcome: | The proposed model solves 1,053 problems spanning 9 mathematical domains . the best-performing model solved only 32.57% of the problems . |
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)
Copied to clipboard
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
| Challenge: | Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks. |
| Approach: | They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions. |
| Outcome: | The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam. |
SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks do not capture the complexity of structured, step-by-step reasoning essential in physics and related domains. |
| Approach: | They propose a large-scale synthetic benchmark of 15K university-level physics problems . they use structured, step-by-step reasoning and executable Python code to produce the ground-truth solution. |
| Outcome: | The proposed model is based on a set of 15K university-level physics problems with three question types. |
Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models (2025.coling-main)
Copied to clipboard
Xinyu Pang, Ruixin Hong, Zhanke Zhou, Fangrui Lv, Xinwei Yang, Zhilong Liang, Bo Han, Changshui Zhang
| Challenge: | Existing large language models (LLMs) fail due to lack of knowledge or incorrect knowledge application. |
| Approach: | They propose a knowledge-augmented framework that constructs a formula set to provide explicit physics knowledge and utilizes checklists to guide effective knowledge application. |
| Outcome: | The proposed framework achieves state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%. |
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks. |
| Approach: | They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters . |
| Outcome: | The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data. |
Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for large reasoning models are saturated by a lack of reliable and verifiable benchmarks. |
| Approach: | They propose a rigorously curated, Olympiad-level math benchmark comprising 350 problems, each with parallel English and Chinese versions. |
| Outcome: | The proposed benchmark unifies two evaluation paradigms and offers 150 problems formalized in Lean 4 for rigorous process-level evaluation. |