SocialGaze: Improving the Integration of Human Social Norms in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasingly, large language models (LLMs) are able to understand and rationalize socially acceptable behaviors, but they are often misaligned with human consensus. |
| Approach: | They propose a multi-step prompting framework that verbalizes a social situation from multiple perspectives before forming a judgment. |
| Outcome: | The proposed framework improves the alignment with human judgments by up to 11 F1 points with the GPT-3.5 model. |
Similar Papers
Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to assess social-pragmatic inference in large language models are inadequacy, and preferential tuning is the best approach. |
| Approach: | They propose to use free-form models' responses as a measure to assess social-pragmatic reasoning and advocate for preference optimization over supervised finetuning (SFT). |
| Outcome: | The proposed model outperforms supervised finetuning (SFT) and offers a near-free launch in pragmatic abilities without compromising general capabilities. |
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely used and engage millions of users from diverse contexts and cultures. |
| Approach: | They propose an evaluation framework to assess LLMs’ cultural adaptability by measuring their ability to judge social acceptability across varying levels of cultural norm specificity. |
| Outcome: | The proposed model shows stronger adaptability to English-centric cultures over those from the Global South. |
Social Intelligence in the Age of LLMs (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a powerful tool for integrating human-like communication and context-aware interactions into artificial systems. |
| Approach: | They propose to introduce and overview different aspects of artificial social intelligence and their relationship with LLMs by introducing scientific methods for evaluating social intelligence in LLM. |
| Outcome: | This tutorial will introduce scientific methods for evaluating social intelligence in LLMs, highlighting the key challenges, and identifying promising research directions. |
SocialEval: Evaluating Social Intelligence of Large Language Models (2025.acl-long)
Copied to clipboard
Jinfeng Zhou, Yuxuan Chen, Yihan Shi, Xuanming Zhang, Leqi Lei, Yi Feng, Zexuan Xiong, Miao Yan, Xunzhi Wang, Yaru Cao, Jianing Yin, Shuai Wang, Quanyu Dai, Zhenhua Dong, Hongning Wang, Minlie Huang
| Challenge: | Existing work on LLMs does not address their social intelligence (SI) and their discrepancy with humans. |
| Approach: | They propose a script-based bilingual SI benchmark that integrates outcome-oriented goal achievement evaluation and process-oriented interpersonal ability evaluation by manually crafting narrative scripts. |
| Outcome: | The proposed model is based on a script-based bilingual evaluation paradigm that integrates outcome- and process-oriented evaluation by manually crafting narrative scripts. |
Measuring Social Norms of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing datasets that evaluate a general understanding of social science are inadequate to understand social norms. |
| Approach: | They propose a multi-agent framework to improve large language models’ ability to understand social norms by comparing them to elementary students. |
| Outcome: | The proposed framework improves large language models to be on par with humans. |
How Inclusively do LMs Perceive Social and Moral Norms? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) are used in decision-making systems and as interactive assistants. |
| Approach: | They propose to prompt 11 LMs on rules-of-thumb and compare their outputs with 100 human annotators. |
| Outcome: | The proposed model is compared with 100 human annotators to find out if they are inclusive of diverse human values. |
Are Large Language Models (LLMs) Good Social Predictors? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can generate human-like responses, but it is unclear how well they work and where the plausible predictions derive from. |
| Approach: | They propose to use LLMs to generate human-like responses by mutability and accessibility of social inputs to perform a social prediction task. |
| Outcome: | The proposed model performs well in three realistic settings and a novel social prediction task. |
Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can lead to undesired consequences when misaligned with human values . previous studies have shown misalignment of LLMs with human value using expert-designed or agent-based emulated bias scenarios . |
| Approach: | They investigate whether large language models (LLMs) are misaligned with human values . they find no significant differences in understanding of HVSB between LLMs . |
| Outcome: | The results show that large language models do not have lower misalignment rates and attack success rates . the study also shows that smaller language models have the ability to explain HVSB . |
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks of social language are lacking for large language models. |
| Approach: | They propose a new benchmark that measures how well large language models understand social language by grouping 58 tasks into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
| Outcome: | The proposed model performs well at 58 tasks that are divided into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
To Mask or to Mirror: Human-AI Alignment in Collective Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used to model and augment collective decision-making. |
| Approach: | They propose a framework for assessing collective alignment using the Lost at Sea social psychology task. |
| Outcome: | The proposed framework compares LLMs with human-AI alignment on the Lost at Sea social psychology task. |