Papers by Youngjae Yu
EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild (2025.findings-naacl)
Copied to clipboard
Junhyeok Kim, Min Soo Kim, Jiwan Chung, Jungbin Cho, Jisoo Kim, Sungwoong Kim, Gyeongbo Sim, Youngjae Yu
| Challenge: | EgoSpeak predicts when an agent should begin speaking based on egocentric streaming video. |
| Approach: | They propose a framework for real-time speech initiation prediction in egocentric streaming video by modeling the conversation from the camera wearer's first-person perspective. |
| Outcome: | The proposed framework outperforms random and silence-based baselines in real time and highlights the importance of multimodal input and context length in effectively deciding when to speak. |
Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making (2025.emnlp-main)
Copied to clipboard
Yejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chan Young Park
| Challenge: | Existing safety evaluations rely on coarse success rates and domain-specific setups, making it difficult to diagnose why and where these models fail. |
| Approach: | They propose a framework for systematically evaluating the physical safety of LLMs in embodied decision making. |
| Outcome: | The proposed framework assesses the physical safety of LLMs in embodied decision making. |
Persona Dynamics: Unveiling the Impact of Persona Traits on Agents in Text-Based Games (2025.acl-long)
Copied to clipboard
| Challenge: | Text-based interactive environments have long presented formidable challenges for AI. |
| Approach: | They propose a method for projecting human personality traits onto agents to guide their behavior and integrate them into their policy-learning pipelines. |
| Outcome: | The proposed method induces personality in a text-based game agent by integrating personality profiles directly into the agent's policy-learning pipeline. |
SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Despite advances in artificial intelligence, building social intelligence remains a challenge. |
| Approach: | They propose a task to explain why people laugh in a video and a dataset to do this. |
| Outcome: | The proposed dataset generates plausible explanations for laughter in video and in-the-wild videos. |
Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Dataset (2024.findings-acl)
Copied to clipboard
Minjin Kim, Minju Kim, Hana Kim, Beong-woo Kwak, SeongKu Kang, Youngjae Yu, Jinyoung Yeo, Dongha Lee
| Challenge: | Existing datasets for conversational recommender systems lack specific user preferences and explanations for recommendations . current datasets lack specific preferences, hindering high-quality recommendations despite advances in large language models . |
| Approach: | They propose to synthesize a conversational recommendation dataset with persona- and knowledge-augmented LLM simulators to address these challenges. |
| Outcome: | The proposed dataset outperforms baselines in human and automatic evaluations. |
Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models lack this active understanding capacity, limiting their applicability in real-world scenarios. |
| Approach: | They propose a benchmark to assess the impact of multimodal inputs on lexical ambiguities. |
| Outcome: | The proposed benchmark assesses the impact of multimodal inputs on lexical ambiguities. |
MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that multimodal, multimodal approaches to lyrics translation are more effective than text-only approaches. |
| Approach: | They propose a multilingual, multimodal benchmark for singable lyrics translation . they propose syllable-constrained audio-video LLM with Chain-of-Thought . |
| Outcome: | The proposed system outperforms text-based models in singability and contextual accuracy. |
Are Any-to-Any Models More Consistent Across Modality Transfers Than Specialists? (2025.acl-long)
Copied to clipboard
| Challenge: | equivariance evaluations uncover weak but observable consistency through structured analyses of the intermediate latent space enabled by multiple editing operations. |
| Approach: | They use a dataset of 1,000 images paired with captions, editing instructions, and Q&A pairs to evaluate cross-modal transfers rigorously. |
| Outcome: | The proposed models do not consistently demonstrate greater cross-modal consistency than specialized models in pointwise evaluations such as cyclic consistency. |
Representation Bending for Large Language Model Safety (2025.acl-long)
Copied to clipboard
Ashkan Yousefpour, Taeheon Kim, Ryan Sungmo Kwon, Seungbeen Lee, Wonje Jeung, Seungju Han, Alvin Wan, Harrison Ngan, Youngjae Yu, Jonghyun Choi
| Challenge: | Existing safety-enhancing techniques, such as fine-tuning with human feedback or adversarial training, are still vulnerable as they address specific threats and fail to generalize across unseen attacks. |
| Approach: | They propose a new approach that disrupts representations underlying harmful behaviors in Large Language Models by using loss-based fine-tuning. |
| Outcome: | The proposed approach outperforms existing methods such as Circuit Breaker, RMU, and NPO with 95% reduction in attack success rates across diverse jailbreak benchmarks. |
How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models (2024.findings-emnlp)
Copied to clipboard
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel
| Challenge: | a growing influx of misinformation across news and social media is hampered by outdated foundation model training data. |
| Approach: | They propose to use large language models to scale up online policing mechanisms . they evaluate foundation model performance without continual updating . |
| Outcome: | The proposed model can improve performance without continual updating . the proposed model improves on two widely used benchmarks . |
Right at My Level: A Unified Multilingual Framework for Proficiency-Aware Text Simplification (2026.acl-long)
Copied to clipboard
| Challenge: | Existing large language model-based readability control methods rely on pre-labeled sentence corpora and primarily target English. |
| Approach: | They propose a framework for adaptive multilingual text simplification without parallel corpora supervision that integrates three reward modules: vocabulary coverage, semantic preservation, and coherence. |
| Outcome: | The proposed framework achieves higher lexical coverage at target proficiency levels while maintaining original meaning and fluency compared to stronger LLMs. |
Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Visual arguments rely on images to persuade viewers to do or believe something . |
| Approach: | They propose three tasks for evaluating visual argument understanding . they use visual premises, commonsense premises and reasoning trees to analyze visual arguments . |
| Outcome: | The proposed tasks evaluate visual argument understanding using a dataset of 1,611 images annotated with 5,112 visual premises (with regions), 5,574 commonsense premises, and reasoning trees connecting them into structured arguments. |
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for learning audio-text connections rely on parallel audio- text data . a new approach allows for the representation of environmental soundscapes without using parallel data - a challenge for many applications . |
| Approach: | They propose a model that induces Audio-Text alignment without using parallel audio-text data. |
| Outcome: | The proposed model outperforms the current state-of-the-art for audio classification tasks with no audio-text data by 2.2% on the ESC50 and US8K tasks. |
Aligning Large Language Models by On-Policy Self-Judgment (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model for on-policy learning. |
| Approach: | They propose a new alignment framework that does on-policy learning and is parameter efficient . they propose Judge-augmented Supervised Fine-Tuning to train a single model to act as a policy and a judge. |
| Outcome: | The proposed framework outperforms baselines in preference benchmarks and rejecting sampling by itself improves performance without additional evaluator. |
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding (2026.acl-long)
Copied to clipboard
Jaehyun Jeon, Min Soo Kim, Janghan Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu
| Challenge: | Recent studies focus on surface-level features, overlooking how design choices influence user behavior at scale. |
| Approach: | They propose a benchmark for multimodal understanding of how UI/UX design affects user behavior built on 300 real-world UI image pairs from industry A/B tests. |
| Outcome: | The proposed benchmarks show that models exhibit limited understanding of the behavioral impact of UI/UX design. |
Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal UNcommonsense (MUN) is a benchmark designed to evaluate models’ ability to handle scenarios that deviate from typical visual or contextual expectations. |
| Approach: | They propose a retrieval-based in-context learning framework that transfers reasoning capabilities from larger models to smaller ones without additional training. |
| Outcome: | The proposed method improves on baseline ICL methods by 8.3% over previous methods. |
VLIS: Unimodal Language Models Guide Multimodal Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models face challenges in tasks that require complex linguistic understanding. |
| Approach: | They propose a framework that combines visual conditioning and linguistic understanding of unimodal text-only language models without further training to improve vision-language models. |
| Outcome: | The proposed framework improves vision-language models on diverse tasks including commonsense understanding and complex text generation. |
Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory (2024.findings-emnlp)
Copied to clipboard
Suyeon Lee, Sunghwan Kim, Minju Kim, Dongjin Kang, Dongil Yang, Harim Kim, Minseok Kang, Dayi Jung, Min Kim, Seungbeen Lee, Kyong-Mee Chung, Youngjae Yu, Dongha Lee, Jinyoung Yeo
| Challenge: | Existing models that use large language models are not available due to ethical concerns, and data privacy concerns are a concern. |
| Approach: | They propose a multi-turn dialogue dataset that emulates real-life counseling interactions using the goal-oriented approach of Cognitive Behavioral Therapy (CBT). |
| Outcome: | The proposed model outperforms other models in counseling skills, highlighting its effectiveness and potential as a counseling agent. |
Investigating Counterfactual Unfairness in LLMs towards Identities through Humor (2026.acl-long)
Copied to clipboard
Shubin Kim, Yejin Son, Junyeong Park, Keummin Ka, Seungbeen Lee, Jaeyoung Lee, Hyeju Jang, Alice Oh, Youngjae Yu
| Challenge: | Large Language Models (LLMs) absorb social and cultural biases embedded in vast web-scale corpora and are increasingly deployed in high-stakes domains such as hiring, education, and law. |
| Approach: | They propose a framework to investigate counterfactual unfairness through humor by observing how the model’s responses change when we swap who speaks and who is addressed while holding other factors constant. |
| Outcome: | The proposed framework covers humor generation refusal, speaker intention inference, and relational/societal impact prediction tasks. |
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (2023.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
| Challenge: | a dataset of 1.5 million conversations distilled from everyday spoken situations is limited in scale due to its associated costs. |
| Approach: | They propose to make SODA a publicly available, million-scale high-quality social dialogue dataset . they contextualize social commonsense knowledge from a knowledge graph to distill broad spectrum of social interactions . |
| Outcome: | The proposed dataset is the first publicly available, million-scale high-quality social dialogue dataset. |
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, Maarten Sap
| Challenge: | Existing dialogue systems fail to respond properly to potentially unsafe user utterances . existing systems either ignore or passively agree with unsafe content . |
| Approach: | They introduce a dataset to teach conversational agents to respond to problematic content following social norms. |
| Outcome: | The proposed dataset shows that ProsocialDialog generates more socially acceptable dialogues than existing models. |
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms (2023.emnlp-main)
Copied to clipboard
Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu
| Challenge: | NormLens is a visual-grounded framework for understanding commonsense norms . state-of-the-art models are not well-aligned with human annotation, we show . |
| Approach: | They propose a visual-grounded framework to study commonsense norms by NormLens . they find that models are not well-aligned with human annotation . |
| Outcome: | The proposed model judgments and explanations are not well-aligned with human annotations. |
Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step (2023.acl-long)
Copied to clipboard
| Challenge: | Symbolic Chain-of-thought Distillation (SCoTD) is a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. |
| Approach: | They propose a method to train a smaller student model on rationalizations from a larger teacher model. |
| Outcome: | The proposed method improves the performance of a student model in supervised and few-shot settings and especially for challenge sets. |
Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues (2025.acl-long)
Copied to clipboard
Youngmin Kim, Jiwan Chung, Jisoo Kim, Sunghyun Lee, Sangkyu Lee, Junhyeok Kim, Cheoljong Yang, Youngjae Yu
| Challenge: | Existing large language models fail to incorporate nonverbal elements into conversational experiences. |
| Approach: | They propose a multimodal language model that generates nonverbal cues alongside text . their dataset is annotated with time-aligned text, facial expressions, and body language . |
| Outcome: | The proposed model generates nonverbal languages and text, corresponding to conversational input. |
Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents (2023.emnlp-main)
Copied to clipboard
Hyungjoo Chae, Yongho Song, Kai Ong, Taeyoon Kwon, Minjin Kim, Youngjae Yu, Dongha Lee, Dongyeop Kang, Jinyoung Yeo
| Challenge: | a human-like chatbot requires commonsense reasoning to comprehend and respond to information . however, identifying and aggregating key evidence within a single hop is a challenge . a knowledge distillation framework is proposed that leverages LLMs as unreliable teachers . |
| Approach: | They propose a framework that leverages large language models as unreliable teachers to facilitate multi-hop reasoning over a dialogue context. |
| Outcome: | The proposed framework leverages LLMs as unreliable teachers and selectively distills consistent and helpful rationales via alignment filters. |
GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance (2026.acl-long)
Copied to clipboard
Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Jiwan Chung, Jisung Kim, Ji Hoon Joung, Youngjae Yu
| Challenge: | Recent advances in multimodal large language models (MLLMs) offer new opportunities for higher-level scene understanding, but they require labor-intensive, expert annotation. |
| Approach: | They propose a dataset that combines 2K human-verified images with 22K image-description pairs to provide a more accurate representation of pedestrian scenes. |
| Outcome: | The proposed dataset improves scalability while maintaining quality. |
Tracing Mathematical Proficiency Through Problem-Solving Processes (2026.findings-acl)
Copied to clipboard
| Challenge: | Knowledge Tracing (KT) models a learner's evolving knowledge state over time, but lacks the rich information embedded in students' problem-solving processes. |
| Approach: | They propose a framework that uses a teacher-student-teacher pipeline to extract students’ Mathematical Proficiency (MP) as intermediate representation. |
| Outcome: | The proposed framework improves the prediction performance of existing KT methods and provides interpretable explanations by explicitly modeling students’ mathematical proficiency. |
C2: Scalable Auto-Feedback for LLM-based Chart Generation (2025.naacl-long)
Copied to clipboard
Woosung Koh, Janghan Yoon, MinHyung Lee, Youngjin Song, Jaegwan Cho, Jaehyun Kang, Taehyeon Kim, Se-Young Yun, Youngjae Yu, Bongshin Lee
| Challenge: | generating high-quality charts with Large Language Models presents significant challenges due to limited data and the high cost of curation. |
| Approach: | They propose a referencefree automatic feedback generator to generate high-quality charts with Large Language Models. |
| Outcome: | The proposed framework outperforms baselines and shows that it significantly improves data diversity. |
Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have influenced the development of video large multimodal models (VLMMs). |
| Approach: | They propose a method that integrates video descriptions as context into a multimodal AI system to enrich the understanding of video content. |
| Outcome: | Empirical evaluations show that the proposed approach outperforms existing approaches for video large multimodal models (VLMMs) |
Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models (2024.emnlp-main)
Copied to clipboard
Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, Jinyoung Yeo
| Challenge: | Prior work has used LLMs to generate programming language and applied external compilers for such tasks. |
| Approach: | They propose a framework that expresses task-level logic with pseudocode and tailors it to each instance and simulates execution of it. |
| Outcome: | The proposed framework outperforms baselines in diverse reasoning tasks. |
Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics (2025.findings-naacl)
Copied to clipboard
Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, Youngjae Yu
| Challenge: | Recent advances in Large Language Models (LLMs) have led to their adaptation as conversational agents. |
| Approach: | They propose a new benchmark that uses 8K multi-choice questions to assess the personality of Large Language Models. |
| Outcome: | The proposed personality test outperforms existing personality tests for LLMs in reliability and validity. |
DUSK: Do Not Unlearn Shared Knowledge (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work suggests that machine learning models are indistinguishable from models trained on retain sets. |
| Approach: | They propose a benchmark to evaluate machine unlearning under realistic knowledge overlap . they construct documents containing both shared and unique knowledge . |
| Outcome: | The proposed model is indistinguishable from a model retrained on the retain set while only forget-specific content is removed. |
Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle when faced with unseen domains or languages. |
| Approach: | They propose a framework that leverages the broad knowledge of an MLLM to generate cross-modal pre-questions (preQs) before retrieval. |
| Outcome: | Experiments show that PREMIR outperforms existing retrievers on out-of-distribution benchmarks, including closed-domain and multilingual settings, outperforming strong baselines across all metrics. |
VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on embodied agents have addressed the importance of exploration in environments where tasks and solutions are not predefined. |
| Approach: | They propose a virtual escape room that evaluates AI models in a dynamic environment . they propose to integrate memory management and reasoning into the simulation . |
| Outcome: | The proposed model improves in dynamic and exploration-driven environments by integrating memory management and reasoning. |
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2022.naacl-main)
Copied to clipboard
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, Yejin Choi
| Challenge: | Existing paradigms for text generation are left-to-right decoding from autoregressive language models. |
| Approach: | They propose a decoding algorithm that incorporates heuristic estimates of future cost that are efficient for large-scale language models. |
| Outcome: | The proposed method outperforms baselines on five generation tasks and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. |