Papers by Juyoung Suk
CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmark datasets for Korean cultural and linguistic knowledge are derived from the English counterparts through translation, so they overlook cultural contexts. |
| Approach: | They propose to use Korean cultural and linguistic intelligence to assess Korean model performance by providing fine-grained annotations of cultural and cultural knowledge. |
| Outcome: | The proposed dataset includes 1,995 QA pairs and is based on 1,992 Korean exams and textbooks. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)
Copied to clipboard
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate . |
| Approach: | They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs. |
| Outcome: | The proposed language model-based evaluation model is more powerful than its predecessor. |
LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent work on LLM-as-a-Judge has reported higher correlations with human judgments due to its static nature. |
| Approach: | They propose a framework that leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. |
| Outcome: | The proposed framework evaluates six models on reasoning, factuality and instruction-following tasks. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |