MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments

Challenge:	Large Language Models (LLMs) have shown remarkable capabilities in environmental perception, reasoning-based decision-making, and simulating complex human behaviors, particularly in interactive role-playing contexts.
Approach:	They propose a framework to assess LLMs' proficiency in portraying advanced human behaviors through murder mystery games using eight intricately crafted scripts.
Outcome:	The framework evaluates LLMs' performance in portraying advanced human behaviors through murder mystery games.

Similar Papers

Overview

Challenge

Approach

Outcome

MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)

Copied to clipboard

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

Language Models

Challenge:	Several new LLMs have been introduced necessitating their evaluation on non-English languages.
Approach:	They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets.
Outcome:	The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages.

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration (2024.emnlp-main)

Copied to clipboard

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, Jiashi Feng

Language Models

Challenge:	Large language models (LLMs) have advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities.
Approach:	They propose a competition-based benchmark framework specifically designed to assess LLMs within multi-agent environments.
Outcome:	The proposed framework enhances the LLMs’ abilities in navigating complex social and cognitive dimensions by over threefold between the strongest and weakest LLM models.

Beyond Blind Following: Evaluating Robustness of LLM Agents under Imperfect Guidance (2026.eacl-long)

Copied to clipboard

Yao Fu, Ran Qiu, Xinhe Wang, Jacob Sansom, Sathvika Ayyappa Prabhu, Huijie Tang, Jaekyeom Kim, Sungryull Sohn, Honglak Lee

Robustness in NLP

Challenge:	Large language models (LLMs) have shown strong capabilities as task-solving agents across interactive domains, but in complex environments, auxiliary guidance may be imperfect.
Approach:	They propose a benchmark to measure the robustness of large language models under imperfect guidance.
Outcome:	The proposed benchmark compared LLM agents in navigation, cooking, and gaming in a variety of environments with auxiliary guidance and noisy or underspecified instructions extracted from demonstrations.

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games (2024.findings-acl)

Copied to clipboard

Dekun Wu, Haochen Shi, Zhiyuan Sun, Bang Liu

Language Models

Challenge:	In this study, we explore the application of Large Language Models (LLMs) in Jubensha, a Chinese detective role-playing game and a novel area in Artificial Intelligence (AI) driven gaming.
Approach:	They propose to use large language models to foster AI agent development in Jubensha, a Chinese detective role-playing game.
Outcome:	The proposed framework enables AI agents to engage in Jubensha games autonomously.

LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments (2024.acl-long)

Copied to clipboard

Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen

Language Models

Challenge:	Existing benchmarks for evaluating large language models use static datasets, leading to data leakage or overlooking the complexities of multi-agent interactions.
Approach:	They propose a framework that evaluates the diverse capabilities of LLM agents in multi-agent dynamic environments.
Outcome:	The proposed framework assesses the diverse capabilities of LLM agents in multi-agent dynamic environments.

BotChat: Evaluating LLMs’ Capabilities of Having Multi-Turn Dialogues (2024.findings-naacl)

Copied to clipboard

Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, Kai Chen

Dialogue Systems & Conversational Agents

Challenge:	Modern Large Language Models (LLMs) facilitate high-quality, multi-turn dialogues with humans, but human-based evaluation of such a capability requires substantial manual effort.
Approach:	They propose to evaluate LLMs' ability to emulate human-like, multi-turn conversations using an LLM-centric approach.
Outcome:	The proposed model emulates human-like, multi-turn conversations using an LLM-centric approach.

A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)

Copied to clipboard

Nader Akoury, Qian Yang, Mohit Iyyer

Dialogue Systems & Conversational Agents

Challenge:	evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge.
Approach:	They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games.
Outcome:	The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players.

Investigating Human and LLMs’ Decisions in Unverifiable Environments: A Case Study with GitHub Activity Overview (2026.findings-acl)

Copied to clipboard

Zheng Jiang, Wei Wang, Gaowei Zhang, Yang Feng, Yi Wang

Reasoning

Challenge:	examining the behaviors of Large Language Models as artificial social actors is underexplored, especially in unverifiable scenarios where conventional benchmarking has little to help improve their abilities.
Approach:	They propose a method to collect, compare, and reason about human and LLMs' decisions in an unverifiable scenario and use it to examine their behaviors.
Outcome:	The proposed method compared human and LLM decisions in an unverifiable scenario on GitHub and found that proprietary LLMs behave more like humans than open-source LLM systems.

CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds (2025.naacl-long)

Copied to clipboard

Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, Ji-Rong Wen

Language Models

Challenge:	Evaluating role-playing capabilities in large language models is challenging due to complex dynamics involved in role-playering.
Approach:	They propose a simulation sandbox that generates situational fine-grained character behavior trajectories to enhance LLM performance.
Outcome:	The proposed model generates situational fine-grained character behavior trajectories to enhance performance.

Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)

Copied to clipboard

Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou

Language Models

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains.
Approach:	They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies.
Outcome:	The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking.