Papers by Hanwool Lee
What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models (2026.findings-acl)
Copied to clipboard
Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Eungyeol Ahn, Jungwhan Kim, Seunghyeok Hong, Youngsook Song
| Challenge: | HAERAE-Vision benchmarks feature clear, explicit prompts but are often informal and underspecified . state-of-the-art models achieve under 50% on original queries, compared to GPT-5 and Gemini 2.5 Pro . |
| Approach: | They propose a benchmark of 653 real-world visual questions from Korean online communities . they find that even state-of-the-art models achieve under 50% on original queries . |
| Outcome: | HAERAE-Vision benchmarks from Korean online communities yield 1,306 query variants . state-of-the-art models achieve under 50% on original queries, compared with smaller models . authors show that query explicitation alone yields 8 to 22 point improvements . |
KMMLU: Measuring Massive Multitask Language Understanding in Korean (2025.naacl-long)
Copied to clipboard
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, Stella Biderman
| Challenge: | Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. |
| Approach: | They propose to use Korean exams to collect 35,030 questions from an expert-level multiple choice model to capture linguistic and cultural aspects of the Korean language. |
| Outcome: | The proposed benchmark is based on 35,030 questions from original Korean exams. |
ML-Promise: A Multilingual Dataset for Corporate Promise Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Promises shape perceptions and drive decisions, but verification of their fulfillment is difficult due to complexity and volume of commitments . authors propose a new approach to verifying promises in environmental, social, and governance reports . complexity of promises, complexity of evidence, difficulty in verifying their fulfillment a pressing need for new approaches . |
| Approach: | They propose a multilingual dataset that includes English, French, Chinese, Japanese, and Korean . they propose ML-Promise to facilitate in-depth verification of corporate promises . |
| Outcome: | The proposed approach includes promise identification, evidence assessment, and evaluation of timing for verification in multiple languages. |
HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (2024.lrec-main)
Copied to clipboard
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, Songseong Kim
| Challenge: | Existing evaluation tools rely on translations of English datasets or translation-specific benchmarks such as WMT 21 to assess large language models. |
| Approach: | They propose a dataset curated to challenge models lacking Korean cultural and contextual depth. |
| Outcome: | The HAE-RAE Bench challenges models lacking Korean cultural and contextual depth by highlighting their aptitude for recalling Korean-specific knowledge and cultural contexts. |