HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (2024.lrec-main)
Copied to clipboard
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, Songseong Kim
| Challenge: | Existing evaluation tools rely on translations of English datasets or translation-specific benchmarks such as WMT 21 to assess large language models. |
| Approach: | They propose a dataset curated to challenge models lacking Korean cultural and contextual depth. |
| Outcome: | The HAE-RAE Bench challenges models lacking Korean cultural and contextual depth by highlighting their aptitude for recalling Korean-specific knowledge and cultural contexts. |
Similar Papers
Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark (2024.acl-long)
Copied to clipboard
Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, Hwalsuk Lee
| Challenge: | Existing benchmarks for evaluating Large Language Models are limited to the English language. |
| Approach: | They introduce the Open Ko-LLM Leaderboard and Ko-H5 Benchmark as tools for evaluating Large Language Models in Korean using private test sets. |
| Outcome: | The proposed evaluation framework is well integrated in the Korean LLM community. |
CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmark datasets for Korean cultural and linguistic knowledge are derived from the English counterparts through translation, so they overlook cultural contexts. |
| Approach: | They propose to use Korean cultural and linguistic intelligence to assess Korean model performance by providing fine-grained annotations of cultural and cultural knowledge. |
| Outcome: | The proposed dataset includes 1,995 QA pairs and is based on 1,992 Korean exams and textbooks. |
Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on linguistic competence of language models with grammatical knowledge. |
| Approach: | They propose to use grammar as a measurable proxy to assess linguistic competence of large language models (LLMs) . |
| Outcome: | The proposed model aims to assess the linguistic competence of large language models (LLMs) and humans in Korean. |
Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance in the legal domain, with GPT-4 even passing the Uniform Bar Exam in the U.S. However their efficacy remains limited for non-standardized tasks and tasks in languages other than English. |
| Approach: | They propose a benchmark for assessing the Korean legal language understanding of LLMs consisting of 7 legal knowledge tasks and 4 legal reasoning tasks. |
| Outcome: | The proposed model passes the Uniform Bar Exam in the U.S. but its performance is limited for non-standardized tasks and tasks in languages other than English. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
KMMLU: Measuring Massive Multitask Language Understanding in Korean (2025.naacl-long)
Copied to clipboard
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, Stella Biderman
| Challenge: | Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. |
| Approach: | They propose to use Korean exams to collect 35,030 questions from an expert-level multiple choice model to capture linguistic and cultural aspects of the Korean language. |
| Outcome: | The proposed benchmark is based on 35,030 questions from original Korean exams. |
Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research has evaluated large language models' cultural knowledge and contextual understanding, reducing their effectiveness in multicultural settings. |
| Approach: | They propose a benchmark to evaluate LLMs' cultural understanding with a focus on Korean superstitions. |
| Outcome: | The proposed benchmark assesses multilingual LLMs in Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. |
Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset (2025.acl-srw)
Copied to clipboard
| Challenge: | Despite advances in LLMs, there are still concerns about their effectiveness with low-resource agglutinative languages compared to English. |
| Approach: | They evaluated 11 LLMs to assess their understanding of Korean sentence endings . they found that explicitly considering linguistic features improved performance . |
| Outcome: | The evaluated LLMs were able to understand Korean sentences better than other languages. |
SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations for Structured Knowledge (SK) understanding are non-rigorous and focus on a single type of SK. |
| Approach: | They propose a structured knowledge understanding benchmark that includes four widely used structured knowledge forms. |
| Outcome: | The proposed benchmark is based on four widely used structured knowledge forms . it includes a question, an answer, positive knowledge units, and noisy knowledge units . |
From KMMLU-Redux to Pro: A Professional Korean Benchmark Suite for LLM Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using Korean expert-level benchmarks, Large Language Models can be developed in real-world scenarios. |
| Approach: | They introduce two Korean expert-level benchmarks that reflect professional knowledge in Korea. |
| Outcome: | The proposed benchmarks represent professional knowledge in Korea. |