Papers by Juhyun Oh
KOLD: Korean Offensive Language Dataset (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent directions for offensive language detection focus on English and do not transfer well to other languages because of cultural and linguistic differences. |
| Approach: | They present a Korean offensive language dataset annotated with offensive language comments . they use the comments as training data for Korean BERT and RoBERTa models . |
| Outcome: | The proposed model improves offensiveness detection, target classification, and span detection while having room for improvement for target group classification and span prediction. |
Are they lovers or friends? Evaluating LLMs’ Social Reasoning in English and Korean Dialogues (2026.acl-long)
Copied to clipboard
Eunsu Kim, Junyeong Park, Juhyun Oh, Kiwoong Park, Seyoung Song, A. Seza Doğruöz, Alice Oh, Najoung Kim
| Challenge: | Existing studies on LLMs' ability to infer social relationships have limited results for Korean and English. |
| Approach: | They propose a social reasoning task based on a 1.1k-dialogue dataset in English and Korean sourced from movie scripts to evaluate LLMs' ability to infer the social relationships between speakers. |
| Outcome: | The proposed task evaluates the ability of LLMs to infer the social relationships between speakers in 1.1k-dialogue datasets in English and Korean. |
FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks lack systematic methods to identify weaknesses in LLMs . Existing methods to evaluate LLM responses to sensitive topics are lacking . |
| Approach: | They propose a FINE-grained response evaluation taxonomy for sensitive topics that breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness. |
| Outcome: | The proposed model outperforms refinement without guidance on Korean-sensitive questions . FINEST significantly improves the model responses across all three categories . |
Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods struggle to capture subtle inconsistencies in large language models. |
| Approach: | They propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity. |
| Outcome: | The proposed framework detects inconsistencies that prior evaluation methods overlook . it captures subtle deviations that real users would encounter . |
RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs’ Contextual Sensitivity (2026.findings-acl)
Copied to clipboard
| Challenge: | a new benchmark measures the contextual sensitivity of large language models in role conflict scenarios . role conflicts are social dilemmas where multiple roles cannot be fulfilled simultaneously . authors: models are forced to arbitrate between dynamic contextual cues and learned preferences . |
| Approach: | They propose a benchmark to measure the contextual sensitivity of large language models in role conflict scenarios. |
| Outcome: | The proposed benchmark measures the contextual sensitivity of large language models in role conflict scenarios. |
Culture is Everywhere: A Call for Intentionally Cultural Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to evaluate cultural alignment of large language models are too trivial and focus on static facts and values. |
| Approach: | They argue for intentionally cultural evaluation: an approach that examines cultural assumptions . they characterize what, how, and circumstances by which culturally contingent considerations arise in evaluation . |
| Outcome: | The authors argue for intentionally cultural evaluation: an approach that examines cultural assumptions embedded in all aspects of evaluation, not just in explicitly cultural tasks. |
Uncovering Factor-Level Preference to Improve Human-Model Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models exhibit tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. |
| Approach: | They propose a framework to uncover and measure factor-level preference alignment of humans and large language models (LLMs) |
| Outcome: | The proposed framework uncovers and measures factor-level preference alignment of humans and large language models. |
OLA: Output Language Alignment in Code-Switched LLM Interactions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs that code-switch between languages often fail to align with user's implicit language expectation, causing responses to be in undesired languages. |
| Approach: | They propose a benchmark to evaluate LLMs’ Output Language Alignment in code-switched interactions. |
| Outcome: | The proposed benchmark evaluates LLMs’ Output Language Alignment in code-switched interactions. |
The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate” (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing studies on using Large Language Models for model evaluation have focused on using LLMs for reference-free evaluation to meet the needs of long-form text evaluation. |
| Approach: | They propose to use Large Language Models (LLMs) for generation tasks to evaluate models. |
| Outcome: | The proposed model evaluations show that LLMs are less faithful to evaluation tasks than open-source models. |