Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research has evaluated large language models' cultural knowledge and contextual understanding, reducing their effectiveness in multicultural settings. |
| Approach: | They propose a benchmark to evaluate LLMs' cultural understanding with a focus on Korean superstitions. |
| Outcome: | The proposed benchmark assesses multilingual LLMs in Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. |
Similar Papers
ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ScholarBench evaluates domain-specific knowledge of large language models (LLMs) prior benchmarks lack the scalability to handle complex academic tasks. |
| Approach: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark is constructed through a three-step process . |
| Outcome: | ScholarBench evaluates the academic reasoning ability of large language models . the benchmark comprises 5,031 examples in Korean and 5,309 examples in English . |
Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) serve diverse global audiences, making it critical for responsible AI deployment across cultures. |
| Approach: | They propose a framework that conceptualizes alignment along three dimensions: Cultural Group, Cultural Elements and Awareness Scope. |
| Outcome: | The proposed framework reveals critical gaps between benchmarks and real-world cultural biases . region dominates cultural group representation, social and political relations dominates coverage . majority of datasets adopt majority-focused Awareness Scope approaches . |
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models (2026.acl-long)
Copied to clipboard
Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui HE, Chenxin Liu, Zhang Li, null Mahongxia, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao
| Challenge: | Existing multilingual evaluation benchmarks neglect cultural nuances and lack language coverage in subjective tasks. |
| Approach: | They propose a framework that categorizes evaluation tasks into three cultural layers and nine cognitive sub-layers. |
| Outcome: | The proposed framework surpasses prior coverage by up to 111% on 20+ LLMs. |
HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (2024.lrec-main)
Copied to clipboard
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, Songseong Kim
| Challenge: | Existing evaluation tools rely on translations of English datasets or translation-specific benchmarks such as WMT 21 to assess large language models. |
| Approach: | They propose a dataset curated to challenge models lacking Korean cultural and contextual depth. |
| Outcome: | The HAE-RAE Bench challenges models lacking Korean cultural and contextual depth by highlighting their aptitude for recalling Korean-specific knowledge and cultural contexts. |
Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Rapid advancements in large language models have highlighted the need for robust evaluation frameworks that assess their core capabilities. |
| Approach: | They propose two benchmarks to assess core capabilities of large language models . current benchmarks for Thai focus mainly on traditional NLP tasks . |
| Outcome: | The proposed benchmarks are based on evaluations of various LLMs with multi-lingual capabilities and are publicly available to encourage further research and development for Thai LLM. |
CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmark datasets for Korean cultural and linguistic knowledge are derived from the English counterparts through translation, so they overlook cultural contexts. |
| Approach: | They propose to use Korean cultural and linguistic intelligence to assess Korean model performance by providing fine-grained annotations of cultural and cultural knowledge. |
| Outcome: | The proposed dataset includes 1,995 QA pairs and is based on 1,992 Korean exams and textbooks. |
Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses (2026.acl-long)
Copied to clipboard
Chongyuan Dai, Yaling Shen, Zihan Gao, Jia Li, Yishun Jiang, Yaxiong Wang, Liu Liu, Zongyuan Ge, Jinpeng Hu
| Challenge: | Culture is a fundamental determinant of human affective processing and affective perceptions are often limited by declarative knowledge or established societal customs. |
| Approach: | They propose a multimodal benchmark that leverages LLM-generated provisional labels to isolate cross-cultural emotional distinctions. |
| Outcome: | The proposed benchmark captures cross-cultural emotional distinctions and derives reliable ground-truth annotations through human evaluation. |
KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are striving to grasp commonsense reasoning, but they are lacking in Korean commons- ense benchmarks. |
| Approach: | They present a fine-grained benchmark dataset focused on Korean commonsense reasoning that includes multiple-choice questions across seven error categories. |
| Outcome: | The proposed datasets show that LLMs struggle with Korean commonsense reasoning . human accuracy benchmarked at approximately 85%, while GPT-4’s performance lags at about 74%, and other LLM models demonstrate an average accuracy of around 42%. |
FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame Semantics (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation paradigms for large language models lack rigorous methods to evaluate cultural alignment . FRAMENET-CULTURES is an open-ended benchmark for evaluating cultural alignment in LLMs . |
| Approach: | They propose a benchmark for evaluating cultural alignment in large language models based on Fillmore-style frame semantics. |
| Outcome: | The proposed benchmark is based on Fillmore-style frame semantics. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |