Papers by Yejin Jeon
PanicToCalm: A Proactive Counseling Agent for Panic Attacks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for training such models are limited due to ethical and logistical issues. |
| Approach: | They propose a dataset that includes high-distress episodes constructed from first-person narratives and structured around the principles of Psychological First Aid. |
| Outcome: | The proposed model outperforms baseline models in counselor-side metrics and client affect improvement. |
Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Visual programming languages (VPLs) allow users to create programs through graphical interfaces, which results in easier accessibility and widespread usage in various domains. |
| Approach: | They propose to train VPLs from user instructions using large language models . they propose to use retrieval-augmented fine-tuning to leverage repetitive use of subroutines . |
| Outcome: | The proposed method outperforms prompting-based methods for LD generation accuracy even with smaller backbone models. |
Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating speech from facial images rely on pre-trained visual encoders and fine-tune them to align with speech embeddings. |
| Approach: | They propose to derive corresponding voices from facial images using face-to-voice synthesis, which derives corresponding voice from facial image. |
| Outcome: | The proposed approach significantly improves face-voice congruence and synthesis stability. |
Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection (2025.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) provides external knowledge for accurate and up-to-date responses, but external knowledge is vulnerable to poisoning and unauthorized injections. |
| Approach: | They propose a Gradient-based Masked Token Probability defense method to detect and filter out adversarially crafted documents by examining gradients of the retriever’s similarity function. |
| Outcome: | Experiments show that the proposed method eliminates over 90% of poisoned content while retaining relevant documents. |
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding (2026.acl-long)
Copied to clipboard
Jaehyun Jeon, Min Soo Kim, Janghan Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu
| Challenge: | Recent studies focus on surface-level features, overlooking how design choices influence user behavior at scale. |
| Approach: | They propose a benchmark for multimodal understanding of how UI/UX design affects user behavior built on 300 real-world UI image pairs from industry A/B tests. |
| Outcome: | The proposed benchmarks show that models exhibit limited understanding of the behavioral impact of UI/UX design. |
Multi-Level Attention Aggregation for Language-Agnostic Speaker Replication (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent advances in speech synthesis research have enabled the generation of natural-sounding speech, which has prompted a notable shift in TTS research towards the synthesis of speech in the voices of both seen and unseen speakers. |
| Approach: | They propose a multi-level attention aggregation approach that probes and amplifies various speaker-specific attributes in a hierarchical manner. |
| Outcome: | The proposed model achieves substantial speaker similarity and generalizes to out-of-domain (OOD) cases. |
Leveraging the Interplay between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation (2024.lrec-main)
Copied to clipboard
| Challenge: | despite recent advances in speech synthesis, the focus of research has been on high-resource languages like English. |
| Approach: | They propose a framework that incorporates modeling of syntactic and acoustic cues associated with pausing patterns. |
| Outcome: | The proposed framework generates natural speech even for longer and intricate out-of-domain sentences, despite training on short audio clips. |
PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing systems produce generic, monotonic responses that lack individuality and fail to adapt to users’ personal attributes. |
| Approach: | They propose a dataset that incorporates user images as part of the persona, enabling personalized responses tailored to user-specific factors such as age or emotional context. |
| Outcome: | The proposed dataset enhances user experience, with personalized responses contributing to a more engaging interaction. |
Audio-Based Linguistic Feature Extraction for Enhancing Multi-lingual and Low-Resource Text-to-Speech (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to synthesize speech for low-resource languages require a substantial amount of source language corpora to generate the linguistic knowledge that can be reused for speech synthesis. |
| Approach: | They propose a method that extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
| Outcome: | The proposed method extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. |
EnSToM: Enhancing Dialogue Systems with Entropy-Scaled Steering Vectors for Topic Maintenance (2025.findings-acl)
Copied to clipboard
| Challenge: | Small large language models (sLLMs) are lightweight and efficient, but struggle to maintain topic consistency in task-oriented dialogue systems. |
| Approach: | They propose an approach to ensure topic consistency in task-oriented dialogue systems by manipulating internal activations during inference. |
| Outcome: | The proposed approach achieves significant performance gain with a relatively small data size compared to fine-tuning approaches. |
MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored the use of large language models (LLMs) in psychotherapy, however text-based cognitive behavioral therapy models struggle with client resistance, which weakens therapeutic alliance. |
| Approach: | They propose a multimodal approach that incorporates nonverbal cues and a synthetic dataset that pairs each client’s statements with corresponding facial images to train vision language models. |
| Outcome: | The proposed approach outperforms existing text-based cognitive behavioral therapy models in managing client resistance and fostering therapeutic alliance. |
Prompt-Guided Selective Masking Loss for Context-Aware Emotive Text-to-Speech (2025.findings-naacl)
Copied to clipboard
| Challenge: | Emotional dialogue speech synthesis (EDSS) aims to generate expressive speech by leveraging the dialogue context between interlocutors. |
| Approach: | They propose a large language model to generate holistic emotion tags based on prior dialogue context and pinpoint key words in the target utterance that align with the predicted emotional state. |
| Outcome: | The proposed method improves emotional expressiveness and facilitates automatic emotion speech generation during inference. |
Difficulty-Controllable Cloze Question Distractor Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality distractors lack adaptability and control over difficulty levels. |
| Approach: | They propose a two-way distractor generation process to generate plausible distractors using an ensemble QA system and a multitask learning strategy to train a difficulty-controllable generation model. |
| Outcome: | The proposed method significantly outperforms GPT-4o in aligning distractor difficulty with human perception. |
Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle when faced with unseen domains or languages. |
| Approach: | They propose a framework that leverages the broad knowledge of an MLLM to generate cross-modal pre-questions (preQs) before retrieval. |
| Outcome: | Experiments show that PREMIR outperforms existing retrievers on out-of-distribution benchmarks, including closed-domain and multilingual settings, outperforming strong baselines across all metrics. |