SocialEval: Evaluating Social Intelligence of Large Language Models (2025.acl-long)
Copied to clipboard
Jinfeng Zhou, Yuxuan Chen, Yihan Shi, Xuanming Zhang, Leqi Lei, Yi Feng, Zexuan Xiong, Miao Yan, Xunzhi Wang, Yaru Cao, Jianing Yin, Shuai Wang, Quanyu Dai, Zhenhua Dong, Hongning Wang, Minlie Huang
| Challenge: | Existing work on LLMs does not address their social intelligence (SI) and their discrepancy with humans. |
| Approach: | They propose a script-based bilingual SI benchmark that integrates outcome-oriented goal achievement evaluation and process-oriented interpersonal ability evaluation by manually crafting narrative scripts. |
| Outcome: | The proposed model is based on a script-based bilingual evaluation paradigm that integrates outcome- and process-oriented evaluation by manually crafting narrative scripts. |
Similar Papers
Social Intelligence in the Age of LLMs (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a powerful tool for integrating human-like communication and context-aware interactions into artificial systems. |
| Approach: | They propose to introduce and overview different aspects of artificial social intelligence and their relationship with LLMs by introducing scientific methods for evaluating social intelligence in LLM. |
| Outcome: | This tutorial will introduce scientific methods for evaluating social intelligence in LLMs, highlighting the key challenges, and identifying promising research directions. |
InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated the potential to mimic human social intelligence, but most studies focus on static self-report or performance-based tests. |
| Approach: | They propose a framework to assess LLMs' ability to understand and manage intentions by mapping their ability to infer the intentions of others in a game setting. |
| Outcome: | The proposed framework assesses LLMs' ability to understand and manage intentions in a game setting. |
SocialGaze: Improving the Integration of Human Social Norms in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasingly, large language models (LLMs) are able to understand and rationalize socially acceptable behaviors, but they are often misaligned with human consensus. |
| Approach: | They propose a multi-step prompting framework that verbalizes a social situation from multiple perspectives before forming a judgment. |
| Outcome: | The proposed framework improves the alignment with human judgments by up to 11 F1 points with the GPT-3.5 model. |
Are Large Language Models (LLMs) Good Social Predictors? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can generate human-like responses, but it is unclear how well they work and where the plausible predictions derive from. |
| Approach: | They propose to use LLMs to generate human-like responses by mutability and accessibility of social inputs to perform a social prediction task. |
| Outcome: | The proposed model performs well in three realistic settings and a novel social prediction task. |
AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios (2025.naacl-long)
Copied to clipboard
Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Large language models are increasingly employed to empower autonomous agents to simulate human behavior. |
| Approach: | They propose to evaluate LLM-driven agents through multi-turn interactions using a bottom-up approach to create diverse social scenarios constructed from extensive scripts. |
| Outcome: | The proposed model evaluates LLM-driven agents through multi-turn interactions emphasizing goal completion and implicit reasoning. |
Co-Eval: Augmenting LLM-based Evaluation with Machine Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs suffer from biases and misalignment due to limited functional understanding and knowledge gaps. |
| Approach: | They introduce a framework that leverages a criteria planner model and optimized machine metrics to enhance the scalability and fairness of LLM-based evaluation. |
| Outcome: | The proposed framework reduces biases and improves alignment with human preferences, with gains of up to 0.324 in Spearman correlation. |
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks of social language are lacking for large language models. |
| Approach: | They propose a new benchmark that measures how well large language models understand social language by grouping 58 tasks into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
| Outcome: | The proposed model performs well at 58 tasks that are divided into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly important for their intelligence evaluation. |
| Approach: | They propose a game theory-based evaluation platform that measures LLMs’ decision-making strategies and social behaviors in classic game-theoretic settings. |
| Outcome: | The proposed system cross-evaluates 15 leading LLMs using leaderboard rankings and scoring mechanisms. |
Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled richer social simulations . however, the role of information asymmetry in these simulations has been overlooked . |
| Approach: | They develop an evaluation framework to simulate social interactions with LLMs in different settings. |
| Outcome: | The proposed framework performs better in unrealistic, omniscient simulation settings but struggles in those with information asymmetry. |