Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study on mental state reasoning in language models relies on a relatively small sample of closed-source LMs. |
| Approach: | They replicate and extend published work on false belief task by assessing LM mental state reasoning behavior across 41 open-weight models. |
| Outcome: | The results show that large LMs show higher sensitivity and predictive power . they also show that humans and LM models show a bias towards attributing false beliefs . |
Similar Papers
Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used for accessing information on the web. |
| Approach: | They conduct experiments with 80 crowdworkers to compare LLMs with search engines . they ask LLM to provide contrastive information to reduce over-reliance on LLM . |
| Outcome: | The results show that LLMs can outperform search engines but not LLM explanations . the study shows that LMS explanations are not reliable replacements for reading retrieved passages compared to search engines alone. |
Towards Reasoning in Large Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | Reasoning is a fundamental aspect of human intelligence that plays a crucial role in many intellectual activities. |
| Approach: | They propose to improve LLMs' ability to elicit reasoning by providing exemplars or prompts to model reasoning. |
| Outcome: | This paper provides a comprehensive overview of the state of knowledge on reasoning in large language models. |
What Are the Odds? Language Models Are Capable of Probabilistic Reasoning (2024.emnlp-main)
Copied to clipboard
Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, Daniel McDuff
| Challenge: | Language models (LMs) are capable of remarkably complex linguistic tasks, but numerical reasoning is an area in which they struggle. |
| Approach: | They evaluate the probabilistic reasoning capabilities of language models using idealized and real-world statistical distributions. |
| Outcome: | The proposed model can make inferences about distributions, even if assumptions are incorrect or misspecified. |
Evaluating Reasoning Models for Queries with Presuppositions (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work notes that large language models fail to challenge erroneous assumptions and can reinforce users’ misinformed opinions. |
| Approach: | They construct queries with varying degrees of presuppositions spanning health, science, and general knowledge and evaluate several widely-deployed models. |
| Outcome: | The proposed models achieve higher accuracy but fail to challenge a large fraction of false presuppositions. |
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth
| Challenge: | a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias. |
| Approach: | They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias. |
| Outcome: | The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning . |
Can Language Models Recognize Convincing Arguments? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have found that large language models can generate persuasive content without engaging in human experimentation. |
| Approach: | They extend a dataset with debates, votes, and user traits to measure LLMs' ability to distinguish between strong and weak arguments, predict stances based on beliefs and demographic characteristics, and determine appeal of argument to individual based upon their traits. |
| Outcome: | The proposed tasks outperform human predictions in detecting convincing arguments in debates, votes, and user traits. |
Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite growing interest in Theory of Mind (ToM) tasks for evaluating language models, little is known about how LMs internally represent mental states of self and others. |
| Approach: | They propose to investigate how LMs internally represent mental states of self and others . |
| Outcome: | The proposed model size and finetuning significantly improve LMs’ internal representations of others’ beliefs, which are structured - not mere by-products of spurious correlations - yet brittle to prompt variations. |
Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) often display unstable behaviors, e.g., hallucinating unsupported premises, overthinking simple tasks, and displaying higher sensitivity to prompt variations. |
| Approach: | They propose a graph-based analytical framework that clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. |
| Outcome: | The proposed framework enables quantitative evaluation of internal reasoning structure and quality beyond conventional metrics and provides practical insights for prompt engineering and cognitive analysis of LLMs. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in the domain of large language models (LLMs) have showcased their capability in executing deductive reasoning tasks. |
| Approach: | They examine inferential strategies employed by large language models through a detailed evaluation of their responses to propositional logic problems. |
| Outcome: | The proposed model shows that it displays reasoning patterns similar to humans, including strategies like supposition following or chain construction. |