Papers by Hwaran Lee
Who Wrote this Code? Watermarking for Code Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detect machine-generated text by embedding watermarks fail to function appropriately in code generation tasks due to the task’s nature of having low entropy. |
| Approach: | They propose a logit-modifying watermark method which enhances detection ability and mitigates code quality degeneration by removing low-entropy segments at generating and detecting watermarks. |
| Outcome: | The proposed method outperforms baseline methods in detecting machine-generated code text while preserving code quality. |
Plug-and-Play Adaptation for Continuously-updated QA (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing tasks to assess LMs’ efficacy as KBs do not adequately consider multiple large-scale updates. |
| Approach: | They propose a task where multiple large-scale updates are made to language models and plug-in modules are used to handle the updates. |
| Outcome: | The proposed method outperforms existing methods on zsRE QA and NQ datasets and is 4x more effective in terms of updates/forgets ratio compared to a fine-tuning baseline. |
AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence (2025.naacl-long)
Copied to clipboard
| Challenge: | AdvisorQA aims to improve LLMs’ capability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. |
| Approach: | They propose a dataset to train LLMs' ability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. |
| Outcome: | The proposed model improves usefulness through automatic metric, GPT-4 and human evaluations, and expands independent evaluation axis to include harmlessness. |
Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding (2025.acl-long)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) are inherently multilingual agents . concerns regarding their safety have emerged . |
| Approach: | They propose a framework to synthesize red-teaming queries and investigate their safety . they demonstrate that the framework outperforms existing red- teaming techniques . |
| Outcome: | The proposed framework outperforms existing red-teaming techniques in the safety domain . it generates code-switching attack prompts in monolingual data . |
MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty (2025.findings-naacl)
Copied to clipboard
| Challenge: | despite advances in large language models, they still produce false but incorrect responses. |
| Approach: | They propose a new benchmark for large language models that requires more than two unambiguous answers . they also assess 5 different uncertainty quantification methods in the presence of data uncertainty. |
| Outcome: | The proposed method fails in multi-answer question answering tasks compared to single-answered questions . entropy- and consistency-based methods effectively estimate model uncertainty, the authors show . |
Drift: Decoding-time Personalized Alignments with Implicit User Preferences (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Drift personalizes large language models at decoding time with implicit user preferences . Unlike traditional Reinforcement Learning from Human Feedback, Drift operates in a training-free manner . |
| Approach: | They propose a framework that personalizes large language models at decoding time with implicit user preferences. |
| Outcome: | The proposed framework personalizes large language models at decoding time with implicit user preferences. |
Query-Efficient Black-Box Red Teaming via Bayesian Optimization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating test cases and querying fail to be query-efficient . generative models can be used for open-domain dialogue, prompt continuation, text-to-image generation . |
| Approach: | They propose a query-efficient method that iteratively finds diverse positive test cases leading to model failures by utilizing user input and past evaluations. |
| Outcome: | The proposed method finds a significantly larger number of diverse positive test cases under limited query budget than baseline methods. |
Critic-Guided Decoding for Controlled Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. |
| Approach: | They propose a method that combines reinforcement learning and weighted decoding to train a critic from reward models. |
| Outcome: | The proposed method generates more coherent and well-controlled texts than previous methods on three controlled generation tasks, topic control, sentiment control, and detoxification. |
KoSBI: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applications (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing research and resources are not readily applicable in South Korea due to the differences in language and culture, both of which significantly affect the biases and targeted demographic groups. |
| Approach: | They propose a social bias dataset of 34k pairs of contexts and sentences in Korean covering 72 demographic groups in 15 categories. |
| Outcome: | The proposed dataset reduces social biases by 16.47%p on average for HyperClova (30B and 82B), and GPT-3. |
TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. |
| Approach: | They propose a method that uses adversarial suffixes to get an answer from a target LLM. |
| Outcome: | The proposed method detects the LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. |
LifeTox: Unveiling Implicit Toxicity in Life Advice (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing safety benchmarks and red teaming prompts fail to capture implicit toxicity in complex real-life advice-seeking scenarios. |
| Approach: | They propose a dataset designed for identifying implicit toxicity within advice-seeking scenarios. |
| Outcome: | The proposed dataset matches or surpasses the zero-shot performance of large language models in toxicity classification tasks. |
TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be used to simulate human behaviors, but point-in-time role-playing is a key component of fandom role-players. |
| Approach: | They propose a benchmark to evaluate point-in-time character hallucination in role-playing LLMs. |
| Outcome: | The proposed method reduces point-in-time character hallucinations effectively by decomposing reasoning steps and using narrative experts. |
ClaimDiff: Comparing and Contrasting Claims on Contentious Issues (2023.findings-acl)
Copied to clipboard
| Challenge: | Using fact verification tasks, however, can not detect subtle differences in factually consistent claims, which might bias the readers. |
| Approach: | They propose a novel dataset that primarily focuses on comparing the nuance between claim pairs. |
| Outcome: | The proposed dataset shows that human-labeled 2,941 claim pairs are weaker than baselines, showing a 19% absolute gap with the baselines. |
SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration (2023.acl-long)
Copied to clipboard
Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Meeyoung Cha, Yejin Choi, Byoungpil Kim, Gunhee Kim, Eun-Ju Lee, Yong Lim, Alice Oh, Sangchul Park, Jung-Woo Ha
| Challenge: | Existing studies focus on coping with social harms that large language models pose . however, discussions on sensitive issues can become toxic even if the users are well-intentioned. |
| Approach: | They propose to use Korean dataset to test whether LLMs can generate offensive content and propagate prejudices. |
| Outcome: | The proposed dataset shows that acceptable response generation improves for HyperCLOVA and GPT-3. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
Masked Summarization to Generate Factually Inconsistent Summaries for Improved Factual Consistency Checking (2022.findings-naacl)
Copied to clipboard
| Challenge: | Abstractive summarization systems generate paraphrases, but they often contain information inconsistent with the source text. |
| Approach: | They propose to generate factually inconsistent summaries using source texts and reference summary with key information masked to train a factual consistency classifier. |
| Outcome: | The proposed method outperforms existing models and shows a competitive correlation with human judgments. |
SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking (P19-1)
Copied to clipboard
| Challenge: | Existing methods to model domain- and slot-dependent belief trackers have difficulty adding new slot-values, resulting in lack of flexibility of domain ontology configurations. |
| Approach: | They propose a model that captures relationships between domain-slot-types and slot-values appearing in utterances through attention mechanisms based on contextual semantic vectors. |
| Outcome: | The proposed model improves performance on two dialog corpora and achieves state-of-the-art accuracy. |
Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Visual dialog is a task of answering questions grounded in an image using dialog history as context. |
| Approach: | They propose a Sparse Graph Learning method to formulate visual dialog as a graph structure learning task. |
| Outcome: | The proposed model outperforms the state-of-the-art models on the VisDial v1.0 dataset. |
Alignment Data Map for Efficient Preference Data Selection and Diagnosis (2026.findings-acl)
Copied to clipboard
| Challenge: | constructing high-quality preference datasets faces scalability challenges due to prohibitive cost and complexity of human annotation. |
| Approach: | They propose a tool to identify and select effective preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches. |
| Outcome: | The proposed tool reduces annotation costs while preserving alignment effectiveness. |
Code-Switching Curriculum Learning for Multilingual Transfer in LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit near human-level performance in various tasks, but performance drops after a handful of high-resource languages due to the imbalance in pre-training data. |
| Approach: | They propose a code-switching curriculum learning model to enhance cross-lingual transfer for LLMs by progressively training models with a curriculum consisting of token-level code-changing, sentence-level codeswitching, and monolingual corpora. |
| Outcome: | The proposed model improves language transfer to Korean, with significant gains in Japanese and Indonesian . the proposed model mitigates spurious correlations between language resources and safety alignment . |
KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) must possess an understanding of the nation’s culture and basic knowledge. |
| Approach: | They propose to construct a national alignment benchmark, KorNAT, which measures the alignment between an LLM and a targeted country from two perspectives: social value alignment and common knowledge alignment. |
| Outcome: | The proposed model passes the national alignment score of 7 LLMs, indicating there is room for improvement. |