Beyond Self-Report: Bridging the Intention-Behavior Gap in Critical Thinking Assessment via Interpretable Multi-Agent System (2026.acl-long)
Copied to clipboard
Zekun Li, Jifan Yu, Haoxuan Li, Ye He, Daniel Zhang-Li, Shangqing Tu, Joy Jia Yin Lim, Yikun Jiang, Jiaxin Yuan, Yu Zhang
| Challenge: | Accurate assessment of critical thinking is limited by the Intention Behavior Gap in psychology . evaluators that measure self-reported competence are limited by multiagent architectures . |
| Approach: | They propose a framework that operationalizes cognitive assessment into an interpretable multi-agent workflow with Assessment Chain-of-Thought. |
| Outcome: | The proposed framework aligns better with human expert ratings than gold-standard inventories on large-scale simulations and human participants. |
Similar Papers
Beyond Self-Reports: Multi-Observer Agents for Personality Assessment in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Self-report questionnaires are used to assess LLM personality traits, but they fail to capture behavioral nuances due to biases and meta-knowledge contamination. |
| Approach: | They propose a multi-observer framework for personality trait assessments in LLM agents that draws on informant-report methods in psychology. |
| Outcome: | The proposed framework combines multiple observers with a subject LLM agent to assess its Big Five personality traits. |
360∘REA: Towards A Reusable Experience Accumulation with 360∘ Assessment for Multi-Agent System (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on optimizing the agent team or employing self-reflection to iteratively solve complex tasks. |
| Approach: | They propose a hierarchical multi-agent framework that uses 360 assessment to accumulate experience through fine-grained assessment. |
| Outcome: | The proposed framework is based on corporate organizational practices and employs a dual-level experience pool for agents to accumulate experience through fine-grained assessment. |
MADRA: Multi-Agent Debate for Risk-Aware Embodied Planning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing safety alignment methods, such as RLHF, fall into a Safety-Utility Trade-off, resulting in severe over-rejection of benign household instructions. |
| Approach: | They propose a meta-cognitive Critical Agent that evaluates peer debates using a structured argumentation framework derived from the Toulmin Model. |
| Outcome: | The proposed architecture outperforms existing systems in the SafeAware-VH benchmark. |
TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing explainability methods for large language models have been limited in capturing interaction-dependent belief dynamics and multi-agent reasoning. |
| Approach: | They propose a tri-view explainability framework that instruments sequential decision making with aligned artifacts. |
| Outcome: | The proposed framework enables analysis of explanation faithfulness, belief dynamics, and evaluator reliability, revealing systematic mismatches between what agents say, what they believe, and what they do. |
Do Androids Question Electric Sheep? A Multi-Agent Cognitive Simulation of Philosophical Reflection on Hybrid Table Reasoning (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing studies have observed that LLMs are wise enough to be thinkers of philosophical reflection. |
| Approach: | They conduct an exploratory simulation to investigate whether LLMs are wise enough to be thinkers of philosophical reflection. |
| Outcome: | The proposed frameworks underperform simpler end-to-end approaches, but they tend to overthink . the results shed light on a fundamental challenge shared by both human and machine intelligence . |
LLM Agents at the Roundtable: A Multi-Perspective and Dialectical Reasoning Framework for Essay Scoring (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework for automated essay scoring is needed to achieve multi-perspective understanding and judgment. |
| Approach: | They propose a roundtable essay scoring framework that performs precise and human-aligned scoring under a zero-shot setting. |
| Outcome: | The proposed framework outperforms previous zero-shot AES approaches by enabling collaboration among agents with diverse evaluation perspectives. |
Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models fail to recognize fallacious reasoning in real-world interactions despite strong performance on static fallacy detection tasks. |
| Approach: | They propose a Chinese benchmark to assess fallacy awareness without explicit cues . they propose 'fate' evaluation framework that assesses fallacy without explicit . |
| Outcome: | The proposed framework assesses fallacy awareness without explicit cues, combining natural dialogue responses and reasoning-based decisions. |
Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models (2026.findings-acl)
Copied to clipboard
Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, Xiaolong Li
| Challenge: | Large language models (LLMs) have evolved from statistical sequence predictors to sophisticated autonomous agents capable of reasoning, planning, and sustaining multi-turn conversa-tions. |
| Approach: | They propose a system that instantiates a "Sentient Agent" that simulates human-like emotional changes and inner thoughts to provide a more realistic evaluation of the model in multi-turn conversations. |
| Outcome: | The proposed framework measures the agent's higher-order social cognition in multi-turn conversations. |
InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated the potential to mimic human social intelligence, but most studies focus on static self-report or performance-based tests. |
| Approach: | They propose a framework to assess LLMs' ability to understand and manage intentions by mapping their ability to infer the intentions of others in a game setting. |
| Outcome: | The proposed framework assesses LLMs' ability to understand and manage intentions in a game setting. |
AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering (2026.acl-long)
Copied to clipboard
Taolin Zhang, Dongyang Li, Chen Chen, Qizhou Chen, Jiuheng Wan, Xiaofeng He, Chengyu Wang, Richang Hong
| Challenge: | Empirical results show that AMATA outperforms baseline approaches, knowledge-augmented frameworks, and LLMs on knowledge-intensive QA benchmarks. |
| Approach: | They propose an Adaptive Multi-Agent Trajectory Alignment framework that integrates external knowledge to improve response interpretability and factual grounding. |
| Outcome: | The proposed framework outperforms baseline approaches, knowledge-augmented frameworks, and LLM-based trajectory systems on five established knowledge-intensive QA benchmarks. |