CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures (2026.eacl-long)
Copied to clipboard
| Challenge: | Game-theoretic interactions between agents with large language models (LLMs) have revealed many emergent capabilities, yet the linguistic diversity of these interactions has not been quantified. |
| Approach: | They propose a metric to quantify the effectiveness of language use within multi-agent systems across different game-theoretic interactions. |
| Outcome: | The proposed metric measures the effectiveness of language use within multi-agent systems across game-theoretic interactions. |
Similar Papers
MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities. |
| Approach: | They propose a competition-based benchmark framework specifically designed to assess LLMs within multi-agent environments. |
| Outcome: | The proposed framework enhances the LLMs’ abilities in navigating complex social and cognitive dimensions by over threefold between the strongest and weakest LLM models. |
LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated emergent common-sense reasoning and Theory of Mind (ToM) capabilities, making them promising candidates for developing coordination agents. |
| Approach: | They propose to use Large Language Models (LLMs) to analyze coordination models in Pure Coordination settings where agents must cooperate to maximize gains. |
| Outcome: | The proposed benchmark evaluates LLMs through two distinct tasks: Agentic Coordination and Coordination Question Answering. |
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly important for their intelligence evaluation. |
| Approach: | They propose a game theory-based evaluation platform that measures LLMs’ decision-making strategies and social behaviors in classic game-theoretic settings. |
| Outcome: | The proposed system cross-evaluates 15 leading LLMs using leaderboard rankings and scoring mechanisms. |
clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents (2023.emnlp-main)
Copied to clipboard
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, David Schlangen
| Challenge: | Recent work suggests large language models can be understood as (simulators of) such agents. |
| Approach: | They propose a method for systematic evaluation of "Situated Language Understanding Agents" they propose implementing a framework for implementing rules to be played in "self-play" |
| Outcome: | The proposed model can be evaluated in game-like settings, the authors show . they show that the model can follow game-play instructions and perform better than existing models . |
LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models use static datasets, leading to data leakage or overlooking the complexities of multi-agent interactions. |
| Approach: | They propose a framework that evaluates the diverse capabilities of LLM agents in multi-agent dynamic environments. |
| Outcome: | The proposed framework assesses the diverse capabilities of LLM agents in multi-agent dynamic environments. |
LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey (2026.findings-acl)
Copied to clipboard
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Jizhou Guo, Yankai Chen, Chunyu Miao, Hoang H Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Hanrong Zhang, Fangxin Wang, Pengfei Zhang, Langzhou He, Yangning Li, Dongyuan Li, Renhe Jiang, Philip S. Yu
| Challenge: | Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents. |
| Approach: | They propose to integrate human-provided information, feedback, or control into the agent system to enhance system performance, reliability, and safety. |
| Outcome: | The proposed systems improve system performance, reliability, and safety by integrating human-provided information, feedback, or control into the agent system. |
MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, Jiaxuan You
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. |
| Approach: | They propose a benchmark to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. |
| Outcome: | The proposed framework measures task completion and quality of collaboration and competition using novel, milestone-based key performance indicators. |
Human Alignment: How Much Do We Adapt to LLMs? (2025.acl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are becoming a common part of our lives, yet few studies have examined how they influence our behavior. |
| Approach: | They propose a cooperative language game in which players aim to converge on a word and play a game in a group. |
| Outcome: | The proposed game shows that humans notice and adapt to differences regardless of whether they are aware they are interacting with an LLM. |
AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios (2025.naacl-long)
Copied to clipboard
Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Large language models are increasingly employed to empower autonomous agents to simulate human behavior. |
| Approach: | They propose to evaluate LLM-driven agents through multi-turn interactions using a bottom-up approach to create diverse social scenarios constructed from extensive scripts. |
| Outcome: | The proposed model evaluates LLM-driven agents through multi-turn interactions emphasizing goal completion and implicit reasoning. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |