Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) perform complex mathematical reasoning in Physics where physical context requires that any symbolic manipulation satisfies complex semantics. |
| Approach: | They systematically remove crucial context from prompts to force instances where model inference may be algebraically coherent, yet unphysical. |
| Outcome: | The proposed models perform poorly in this domain, and their reasoning is not physics-informed. |
Similar Papers
Fine-Grained Detection of Context-Grounded Hallucinations Using LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing representations of hallucinations limit the types of errors that can be expressed, so we propose a new representation based on free-form textual descriptions, capturing the full range of possible errors. |
| Approach: | They propose a benchmark for localizing hallucinations using LLMs with a human annotation of over 1,000 examples and a protocol to verify its quality in a humans evaluation. |
| Outcome: | The proposed representation captures the full range of possible errors, and the best model achieves an F1 score of 0.67. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
NEWTON: Are Large Language Models Capable of Physical Reasoning? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have been shown to encapsulate syntactic, semantic, word sense, and common-sense knowledge, but limited exploration of their physical reasoning abilities has been conducted. |
| Approach: | They propose a repository and benchmark to evaluate LLMs' physical reasoning skills . they use a pipeline to generate a variant of the benchmark customized to the objects and attributes relevant for their application. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of language models across reasoning tasks. |
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. |
| Approach: | They propose a commonsense reasoning dataset with dense annotations that allows multi-tiered evaluation of machines’ reasoning process. |
| Outcome: | The proposed model can achieve high end performance but struggle to support predictions with valid supporting evidence. |
How Well Do Large Language Models Truly Ground? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing research defines “grounding” as having the correct answer, which does not ensure the reliability of the entire response. |
| Approach: | They propose a stricter definition of grounding: fully utilizes the necessary knowledge from the provided context and stays within the limits of that knowledge. |
| Outcome: | The proposed model can be ground on external contexts and maintain its correct answer. |
Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models struggle with semantic inertia, a problem that is often attributed to natural language encoding, which entangles descriptive semantics and logical rules, leading to persistent hallucinations of familiar physics despite explicit contradictory rules. |
| Approach: | They propose a framework that decouples logical dynamics from visual priors via amortized theory induction and counterfactual contrastive alignment. |
| Outcome: | The proposed framework outperforms expensive inference-time search methods in both efficiency and accuracy. |
STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing (2025.findings-acl)
Copied to clipboard
| Challenge: | Advances in large language models have spurred research into enhancing their reasoning capabilities, particularly in math-rich STEM documents. |
| Approach: | They propose a benchmark dataset to evaluate LLMs’ reasoning abilities on math symbols within contextual scientific text. |
| Outcome: | The proposed dataset demonstrates that state-of-the-art LLMs achieve an average accuracy of 20-60% under in-context learning and 50-60% with fine-tuning, highlighting a substantial gap in their ability to classify mathematical symbols. |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics (2022.lrec-1)
Copied to clipboard
| Challenge: | physicists use mathematics to reason and explain, separates their field from other disciplines, including mathematics. |
| Approach: | They present a dataset to evaluate the performance of language models in physics . they find that language models are challenged by coherence related tasks in physicists . |
| Outcome: | The proposed models are able to perform well on coherence-related tasks even when trained on natural language objectives. |
Born Pragmatic, Trained to Hallucinate? Quantifying the Origins of Contextual Bias in LLMs via the PaCE Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel at capturing communicative intent, but they have a side effect: pragmatic hallucination. |
| Approach: | They propose a benchmark to quantify the impact of pragmatic hallucination on large language models . they propose RLHF and SFT to induce a strong tendency for pragmatic over-attribution . |
| Outcome: | The proposed model outperforms existing models in predicting pragmatic hallucinations . the evaluations show that current alignment paradigms lack precise control over pragmatic boundaries . |