CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Cultural competence is defined as the ability to understand and adapt to multicultural contexts. |
| Approach: | They propose a framework that uses a hierarchical multilingual taxonomy and a Retrieval-Augmented Generation to synthesize culturally relevant question-answer pairs. |
| Outcome: | The proposed framework contains a hierarchical multilingual taxonomy covering 12 primary and 130 secondary topics and a Retrieval-Augmented Generation (RAG)-based methodology leveraging factual knowledge to synthesize culturally relevant question-answer pairs. |
Similar Papers
CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)
Copied to clipboard
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
| Challenge: | CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Approach: | They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Outcome: | The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East. |
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models encode rich cultural knowledge, but it remains mostly implicit and unstructured, limiting its interpretability and use. |
| Approach: | They propose an iterative framework for constructing a Cultural Commonsense Knowledge Graph using a prompt-based framework. |
| Outcome: | The proposed framework improves cultural reasoning and story generation on non-English cultures. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame Semantics (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for large language models lack rigorous methods to evaluate cultural alignment . FRAMENET-CULTURES is an open-ended benchmark for evaluating cultural alignment in LLMs . |
| Approach: | They propose a benchmark for evaluating cultural alignment in large language models based on Fillmore-style frame semantics. |
| Outcome: | The proposed benchmark is based on Fillmore-style frame semantics. |
Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness (2025.findings-acl)
Copied to clipboard
Bryan Li, Fiona Luo, Samar Haider, Adwait Agashe, Siyu Li, Runqi Liu, Miranda Muqing Miao, Shriya Ramakrishnan, Yuan Yuan, Chris Callison-Burch
| Challenge: | Existing approaches to retrieval-augmented generated (RAG) can be useful in multilingual settings, but they also introduce biases in the retrieved documents. |
| Approach: | They propose a dataset of territorial disputes paired with retrieved Wikipedia documents in 49 languages to evaluate cross-lingual robustness. |
| Outcome: | The proposed paradigm helps mitigate hallucinations of large language models (LLMs). |
Towards Geo-Culturally Grounded LLM Generations (2025.acl-short)
Copied to clipboard
| Challenge: | Contemporary large language models (LLMs) are pretrained on huge corpora of natural language text and fine-tuned using human feedback to improve their quality. |
| Approach: | They compare the performance of standard LLMs, LLM augmented with retrievals from a bespoke knowledge base and LLM with retrieval from . a web search on multiple cultural awareness benchmarks. |
| Outcome: | The retrieval augmented generation and search grounding techniques improve LLMs' ability to display familiarity with various national cultures on cultural awareness benchmarks. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models (2026.acl-long)
Copied to clipboard
Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui HE, Chenxin Liu, Zhang Li, null Mahongxia, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao
| Challenge: | Existing multilingual evaluation benchmarks neglect cultural nuances and lack language coverage in subjective tasks. |
| Approach: | They propose a framework that categorizes evaluation tasks into three cultural layers and nine cognitive sub-layers. |
| Outcome: | The proposed framework surpasses prior coverage by up to 111% on 20+ LLMs. |
NativQA: Multilingual Culturally-Aligned Natural Query for LLMs (2025.findings-acl)
Copied to clipboard
Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, Firoj Alam
| Challenge: | Existing frameworks for QA datasets lack regional specificity and cultural specificity. |
| Approach: | They propose a framework to quench native language QA datasets in native languages for LLM evaluation and tuning. |
| Outcome: | The proposed framework is scalable, language-independent and can be used to build culturally and regionally aligned QA datasets in native languages. |
Scaling Cultural Resources for Improving Generative Models (2026.findings-eacl)
Copied to clipboard
Hayk Stepanyan, Aishwarya Verma, Andrew Zaldivar, Rutledge Chin Feman, Erin MacMurray van Liemt, Charu Kalia, Vinodkumar Prabhakaran, Sunipa Dev
| Challenge: | generative models have been known to have reduced performance in different global cultural contexts and languages. |
| Approach: | They construct a pipeline to collect and contribute culturally salient, multilingual data . they argue such data can assess the state of the global applicability of generative AI models . |
| Outcome: | The proposed pipeline can assess the state of the global applicability of our models and improve upon cross-cultural gaps. |