FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame Semantics (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for large language models lack rigorous methods to evaluate cultural alignment . FRAMENET-CULTURES is an open-ended benchmark for evaluating cultural alignment in LLMs . |
| Approach: | They propose a benchmark for evaluating cultural alignment in large language models based on Fillmore-style frame semantics. |
| Outcome: | The proposed benchmark is based on Fillmore-style frame semantics. |
Similar Papers
Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) serve diverse global audiences, making it critical for responsible AI deployment across cultures. |
| Approach: | They propose a framework that conceptualizes alignment along three dimensions: Cultural Group, Cultural Elements and Awareness Scope. |
| Outcome: | The proposed framework reveals critical gaps between benchmarks and real-world cultural biases . region dominates cultural group representation, social and political relations dominates coverage . majority of datasets adopt majority-focused Awareness Scope approaches . |
MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding Evaluation (2026.acl-long)
Copied to clipboard
Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao, Chaojun Wang, Long Li, Rui Liu, Huiyao Liu, Koji Inoue, Ryuichi Sumida, Tatsuya Kawahara, Fan Xu, Lingyu Ye, Wei Tian, Dongjun Kim, Jimin Jung, Jaehyung Seo, Nadya Yuki Wangsajaya, Pham Minh Duc, Ojasva Saxena, Palash Nandi, Xiyan Tao, Wiwik Karlina, Tuan Luong, Keertana Arun Vasan, Roy Ka-Wei Lee, Nancy F. Chen
| Challenge: | Existing models lack cultural alignment across modalities and languages . a new framework to assess cultural awareness across linguistics and languages is needed . |
| Approach: | They propose a framework that integrates tri-modally aligned cultural benchmarks and a five-dimensional evaluation protocol to assess cross-country awareness disparities. |
| Outcome: | The proposed framework assesses cultural awareness disparities across modalities and languages . it is the first dataset aligned at the input level across text, image, and speech . |
Can LLMs Express Personality Across Cultures? Introducing CulturalPersonas for Evaluating Trait Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have explored personality evaluation of LLMs, but they largely overlook the interplay between culture and personality. |
| Approach: | They propose a large-scale benchmark for evaluating LLMs’ personality expression in culturally grounded, behaviorally rich contexts. |
| Outcome: | The proposed benchmark improves alignment with country-specific human personality distributions and elicits more expressive, culturally coherent outputs compared to existing benchmarks. |
Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses (2026.acl-long)
Copied to clipboard
Chongyuan Dai, Yaling Shen, Zihan Gao, Jia Li, Yishun Jiang, Yaxiong Wang, Liu Liu, Zongyuan Ge, Jinpeng Hu
| Challenge: | Culture is a fundamental determinant of human affective processing and affective perceptions are often limited by declarative knowledge or established societal customs. |
| Approach: | They propose a multimodal benchmark that leverages LLM-generated provisional labels to isolate cross-cultural emotional distinctions. |
| Outcome: | The proposed benchmark captures cross-cultural emotional distinctions and derives reliable ground-truth annotations through human evaluation. |
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models encode rich cultural knowledge, but it remains mostly implicit and unstructured, limiting its interpretability and use. |
| Approach: | They propose an iterative framework for constructing a Cultural Commonsense Knowledge Graph using a prompt-based framework. |
| Outcome: | The proposed framework improves cultural reasoning and story generation on non-English cultures. |
Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues (2026.acl-long)
Copied to clipboard
Muhammad Dehan Al Kautsar, Saeed Almheiri, Momina Ahsan, Bilal Elbouardi, Younes Samih, Sarfraz Ahmad, Amr Keleg, Omar El Herraoui, Kareem Elzeky, Abed Alhakim Freihat, Mohamed Anwar, Zhuohan Xie, Junhong Liang, Mohammad Rustom Al Nasar, Preslav Nakov, Fajri Koto
| Challenge: | Most benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking cultural nuances that naturally arise in dialogues. |
| Approach: | They propose a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both Modern Standard Arabic (MSA) and each country’s respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. |
| Outcome: | The proposed model performs worse on all three tasks than the MSA benchmark. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
AlignCultura: Towards Culturally Aligned Large Language Models? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks represent early steps toward cultural alignment, yet no benchmarks currently enables systematic evaluation of cultural alignment in line with UNESCO’s principles of cultural diversity w.r.t HHH paradigm. |
| Approach: | Align-Cultura aims to evaluate cultural alignment in large language models . it uses a Query Construction pipeline to reclassify prompts and expand underrepresented domains . response generation pairs prompts with culturally grounded responses . |
| Outcome: | Empirically, culturally fine-tuned models improve joint HHH by 4%–6%, reduce cultural failures by 18%, achieve 10%–12% efficiency gains, and limit leakage to 0.3%. |
Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede’s Cultural Dimensions (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are deployed in many countries, but they fail to account for cultural variances among their potential users. |
| Approach: | They propose to use Hofstede’s cultural dimension framework to quantify cultural alignment using latent variable analysis to evaluate large language models against cultural dimensions of regions like the United States, China, and Arab countries. |
| Outcome: | The proposed model is compared against LLMs in the United States, China, and Arab countries and demonstrates that all models struggle to grasp cultural values, while GPT-4 shows a unique capability to adapt to cultural nuances, particularly in Chinese settings. |
Towards Measuring and Modeling “Culture” in LLMs: A Survey (2024.emnlp-main)
Copied to clipboard
Muhammad Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Aji, Jacki O’Neill, Ashutosh Modi, Monojit Choudhury
| Challenge: | Existing models are biased towards Western, Anglocentric or American cultures, a problem that is arguably detrimental to the performance of LLMs. |
| Approach: | They analyze more than 90 recent papers that aim to study cultural representation and inclusion in large language models. |
| Outcome: | The proposed models are biased towards Western, Anglocentric or American cultures, despite their diversity and their robustness. |