Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models are biased towards a Western and Anglo-centric worldview. |
| Approach: | They propose to extend the Octopus test to measure "cultural awareness" they argue that cultural awareness is needed for AI systems to be useful across cultures . |
| Outcome: | The proposed method argues that cultural awareness is not cultural knowledge, but meta-cultural competence . the proposed method is based on the octopus test, which shows it is impossible to learn meaning from real-world concepts without knowing intent and meaning . |
Similar Papers
Towards Measuring and Modeling “Culture” in LLMs: A Survey (2024.emnlp-main)
Copied to clipboard
Muhammad Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Aji, Jacki O’Neill, Ashutosh Modi, Monojit Choudhury
| Challenge: | Existing models are biased towards Western, Anglocentric or American cultures, a problem that is arguably detrimental to the performance of LLMs. |
| Approach: | They analyze more than 90 recent papers that aim to study cultural representation and inclusion in large language models. |
| Outcome: | The proposed models are biased towards Western, Anglocentric or American cultures, despite their diversity and their robustness. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
SocialCC: Interactive Evaluation for Cultural Competence in Language Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have evaluated cultural knowledge of large language models, but they fail to assess dynamic cultural competence. |
| Approach: | They propose a benchmark to assess cultural competence through intercultural scenarios that span 60 countries across six continents. |
| Outcome: | The proposed benchmark measures the ability of large language models to apply cultural knowledge effectively in cross-cultural interactions. |
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. |
| Approach: | They conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. |
| Outcome: | The language used to query the LLMs can impact their performance on cultural-related tasks. |
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models encode rich cultural knowledge, but it remains mostly implicit and unstructured, limiting its interpretability and use. |
| Approach: | They propose an iterative framework for constructing a Cultural Commonsense Knowledge Graph using a prompt-based framework. |
| Outcome: | The proposed framework improves cultural reasoning and story generation on non-English cultures. |
Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing large language models struggle to capture some language styles without fine-tuning. |
| Approach: | They propose to meta-trained LLMs based on representative lexicons to recognize new styles they have not been fine-tuned on. |
| Outcome: | The proposed method improves zero-shot transfer across styles on 13 established and 63 novel tasks generated with LLMs. |
From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual and cross-cultural WAT reveal how culture modulates perceptual and interactive patterns. |
| Approach: | They propose to embed cultural-specific semantic associations directly within large language models (LLMs) to address cultural preference. |
| Outcome: | The proposed model significantly improves cross-cultural alignment, capturing diverse semantic associations. |
Global Gallery: The Fine Art of Painting Culture Portraits through Multilingual Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | This study examines the ability of Large Language Models to encapsulate cultural nuances across diverse linguistic landscapes. |
| Approach: | They examine the efficacy of language-specific instruction tuning and the impact of pretraining on dominant language data in Large Language Models. |
| Outcome: | The findings highlight a nuanced landscape, with inconsistencies and biases, particularly in non-Western cultures. |
Evaluating Cultural and Social Awareness of LLM Web Agents (2025.findings-naacl)
Copied to clipboard
Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu
| Challenge: | Existing benchmarks often overlook cultural and social awareness . current evaluations focus on task completion, often ignoring the diverse cultural and socio-cultural backgrounds. |
| Approach: | They propose a benchmark to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. |
| Outcome: | The proposed framework evaluates LLM agents’ ability to detect and appropriately respond to norm-violating user queries and observations across two web-based tasks. |
Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) serve diverse global audiences, making it critical for responsible AI deployment across cultures. |
| Approach: | They propose a framework that conceptualizes alignment along three dimensions: Cultural Group, Cultural Elements and Awareness Scope. |
| Outcome: | The proposed framework reveals critical gaps between benchmarks and real-world cultural biases . region dominates cultural group representation, social and political relations dominates coverage . majority of datasets adopt majority-focused Awareness Scope approaches . |