NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding (2024.findings-emnlp)
Copied to clipboard
Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, Yangqiu Song
| Challenge: | Theory of mind evaluations currently focus on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. |
| Approach: | They propose a benchmark to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states. |
| Outcome: | The proposed benchmark builds upon the Belief-Desire-Intention theory and conducts the necessary empirical experiments to evaluate large language models. |
Similar Papers
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations for theory of mind (ToM) use passive narratives that lack interactivity. |
| Approach: | They propose a benchmark to stress-test ToM within information-asymmetric conversational contexts via question answering. |
| Outcome: | The proposed benchmark is challenging for state-of-the-art language models, which perform significantly worse than humans even with chain-of thought reasoning or fine-tuning. |
OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing N-ToM benchmarks lack ambiguous and artificial narratives, lack of personality traits and preferences, and limited diversity in the questions posed. |
| Approach: | They propose a benchmark to assess Neural Theory-of-Mind (N-ToM) with longer and clearer narrative stories, characters with explicit personality traits, actions triggered by character intentions, and questions designed to challenge LLMs’ abilities of modeling characters’ mental states. |
| Outcome: | The proposed test aims to assess the performance of LLMs in the physical and psychological worlds. |
Theory of Mind in Large Language Models: Assessment and Enhancement (2025.acl-long)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is a cornerstone of human social intelligence . Large Language Models (LLMs) are increasingly integrated into daily life . |
| Approach: | They analyze evaluation benchmarks and enhancement strategies to evaluate LLMs' ToM capabilities. |
| Outcome: | The proposed and widely used story-based benchmarks and enhancement strategies are used to evaluate LLMs' ToM capabilities. |
Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. |
| Approach: | They propose a higher-order theory of mind benchmark and introduce a new deception mechanism to evaluate ToM reasoning. |
| Outcome: | The proposed benchmarks show that the LLMs are not performing well on higher-order tasks. |
XToM: Exploring the Multilingual Theory of Mind for Large Language Models (2026.acl-long)
Copied to clipboard
Chunkit Chan, Yauwai Yim, Hongchuan Zeng, Zhiying Zou, Xinyuan Cheng, Zhifan Sun, Zheye Deng, Kawai Chung, Yuzhuo Ao, Fan Yixiang, Cheng Jiayang, Ercong Nie, Ginny Wong, Helmut Schmid, Hinrich Schuetze, Simon See, Yangqiu Song
| Challenge: | Existing evaluations of ToM in LLMs are limited to English, neglecting the linguistic diversity that shapes human cognition. |
| Approach: | They propose a multilingual benchmark that evaluates ToM across five languages . they find that models excel in multilingual language understanding, but their ToM performance varies across languages. |
| Outcome: | The proposed benchmark evaluates LLMs across five languages and incorporates diverse task scenarios. |
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models (2024.eacl-long)
Copied to clipboard
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz
| Challenge: | Recent work suggests that Large Language Models (LLMs) exhibit Neural Theory-of-Mind (N-ToM) however, prior work reached conflicting conclusions regarding those abilities. |
| Approach: | They examine the extent of Large Language Models’ N-ToM abilities through an extensive evaluation of 6 tasks and find that LLMs struggle with adversarial examples . |
| Outcome: | The proposed metrics show that LLMs exhibit certain N-ToM abilities, but this behavior is far from robust. |
GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs (2026.acl-long)
Copied to clipboard
Weidong Tang, Jierui Li, Yueling Hou, Zihan Mei, Can Zhang, Xinyan Wan, Zhiyuan Liang, Pengfei Zhou, Yang You, Wangbo Zhao
| Challenge: | Existing models for general intelligence fail to model how mental states interact and crystallize into group-level outcomes. |
| Approach: | They propose a multimodal benchmark for group-level Theory of Mind (ToM) to probe nonlinear collective behavior. |
| Outcome: | The proposed model performs significantly below human levels, exposing blind spots in modeling social structures and nonlinear collective behavior. |
Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground (2024.findings-acl)
Copied to clipboard
Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow
| Challenge: | Existing benchmarks for theory of mind (ToM) use synthetic data, which can misalign with human behavior. |
| Approach: | They propose a question-answer benchmark based on naturally occurring spoken dialogs to evaluate theory of mind capabilities of language models. |
| Outcome: | The proposed dataset shows that LMs struggle to demonstrate theory of mind (ToM) . |
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent inquiries reveal a lack of robust ToM in large language models . current models focus on different aspects of ToM and are prone to shortcuts and data leakage. |
| Approach: | They propose to taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. |
| Outcome: | The proposed model breaks ToM into individual components and treats LLMs as agents physically and socially situated in interactions with humans. |
Machine Theory of Mind Needs Machine Validation (2025.findings-acl)
Copied to clipboard
| Challenge: | In recent years there has been an explosion of interest in studying the extent to which language models (LMs) display a theory of mind (ToM) despite the growth of evaluation tools, the extent of evidence for ToM remains unclear. |
| Approach: | They conduct a survey of 16 recent studies aimed at measuring ToM in language models and found that only half do so for patterns only a machine might exploit. |
| Outcome: | The results show that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data. |