Papers by Andrew Lee
Towards Algorithmic Fidelity: Mental Health Representation across Demographics in Synthetic vs. Human-generated Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Synthetic data generation has the potential to impact domains with scarce data, but we need to understand how different demographics are represented in it. |
| Approach: | They develop a procedure to generate depression data using GPT-3 and analyze it to uncover the types of stressors it assigns to demographic groups. |
| Outcome: | The proposed procedure produces depression data using GPT-3, and compares it to a human-generated dataset. |
Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Typically, creating verbal cues requires extensive human effort and is quite time-consuming. |
| Approach: | They propose a method for overgenerating and ranking verbal cues by prompting large language models to generate them and ranking them according to psycholinguistic measures and takeaways from a pilot user study. |
| Outcome: | The proposed method is comparable to human-generated mnemonics in imageability, coherence, and perceived usefulness, but there remains room for improvement due to the diversity in background and preference among language learners. |
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
PhoniTale: Phonologically Grounded Mnemonic Generation for Typologically Distant Language Pairs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vocabulary acquisition is a challenge for second-language learners when learning typologically distant languages such as English and Korean, where phonological and structural mismatches complicate vocabulary learning. |
| Approach: | They propose a cross-lingual mnemonic generation system that performs IPA-based phonological adaptation and syllable-aware alignment to retrieve L1 keyword sequence and uses LLMs to generate verbal cues. |
| Outcome: | The proposed system outperforms human-written and automated mnemonics in a short-term recall test with human participants and achieves quality comparable to human-writing mnms. |
Rich Character-Level Information for Korean Morphological Analysis and Part-of-Speech Tagging (C18-1)
Copied to clipboard
| Challenge: | Korean is a highly agglutinative, character-rich language, requiring dictionary-less morphological analysis . a novel model can perform morphology and part-of-speech tagging without prior knowledge . |
| Approach: | They propose a multi-stage action-based model that performs morphological transformation and part-of-speech tagging using arbitrary units of input. |
| Outcome: | The proposed model achieves state-of-the-art word and sentence-level tagging accuracy with Korean corpus. |
Empathy Identification Systems are not Accurately Accounting for Context (2023.eacl-main)
Copied to clipboard
| Challenge: | Empathy is a fundamental phenomenon that allows us to better communicate and relate with others. |
| Approach: | They propose a simple model that checks if an input utterance is similar to a small set of empathetic examples, but does not consider dialogue context. |
| Outcome: | The proposed model outperforms state-of-the-art models on benchmarks and empathetic rationale extraction benchmarks. |
Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Health (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing statistical models are not explainable, struggle in low-resource scenarios and cannot be reused for multiple tasks. |
| Approach: | They propose a micromodel architecture that embeds domain knowledge and provides explanations throughout the model’s decision process. |
| Outcome: | The proposed model is validated on depression classification, PTSD classification, and suicidal risk assessment tasks. |
Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for mnemonic generation in Japanese are limited in their interpretability due to script differences. |
| Approach: | They propose a method that models the mnemonic construction process as driven by common rules. |
| Outcome: | The proposed method performs well in the cold-start setting for new learners while providing insight into the mechanisms behind effective mnemonic creation. |
Eeyore: Realistic Depression Simulation via Expert-in-the-Loop Supervised and Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been explored for mental healthcare training and therapy client simulation, but they fail to authentically capture diverse client traits and psychological conditions. |
| Approach: | They propose an 8B model optimized for realistic depression simulation with expert input at every stage. |
| Outcome: | The model outperforms GPT-4o in linguistic authenticity and profile adherence. |
Outlier Detection for Improved Data Quality and Diversity in Dialog Systems (N19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, Jason Mars
| Challenge: | Existing methods to detect outliers in text have been neglected in NLP . outlier detection is a problem in dialog systems where text is often no more than a few sentences in length. |
| Approach: | They propose a technique that uses sentence embeddings to detect outliers in short texts using neural sentence embeds and distance-based outlier detection. |
| Outcome: | The proposed technique detects outliers in a corpus of short texts while generating highly diverse corpora that produce more robust intent classification and slot-filling models. |
Event-Event Relation Extraction using Probabilistic Box Embedding (2022.acl-short)
Copied to clipboard
| Challenge: | Existing frameworks of event relation extraction do not guarantee coherence between different relation types, such as anti-symmetry. |
| Approach: | They propose to modify existing ERE framework to guarantee coherence by representing each event as a box representation without applying explicit constraints. |
| Outcome: | The proposed model shows stronger conjunctive constraint satisfaction compared to previous models with constraint injection. |
How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a popular choice of safety fine-tuning algorithms, but prior explanations of its effects only account for dampened toxic neurons in the MLP layers. |
| Approach: | They analysed four language models and found that toxic neurons only account for 2.5% to 24% of DPO’s effects across models. |
| Outcome: | The proposed method outperforms DPO in reducing toxicity while preserving perplexity, without requiring any weight updates. |
Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that using only bi-encoders as an intermediate reranker can improve top-1 accuracy with negligible slowdown (7%). |
| Approach: | They propose a framework that compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other. |
| Outcome: | The proposed framework compares a query and multiple embeddings of similar candidates through shallow self-attention layers, delivering rich representations contextualized to each other. |
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021.emnlp-main)
Copied to clipboard
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, Andrew McCallum
| Challenge: | Using human-labeled examples, case-based reasoning can solve complex problems from scratch . case-Based reasoning is a paradigm that is used to solve complex problem . |
| Approach: | They propose a neuro-symbolic CBR approach for question answering over large knowledge bases. |
| Outcome: | The proposed approach outperforms the current state of the art on a CWQ dataset by 11% on accuracy. |
Machine Reading Comprehension using Case-based Reasoning (2023.findings-emnlp)
Copied to clipboard
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
| Challenge: | Current state-of-the-art machine readers do not support case-based reasoning . |
| Approach: | They propose a method that extracts a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers. |
| Outcome: | The proposed method outperforms baselines on NaturalQuestions and NewsQA by 11.5 and 8.4 EM. |
A Comparative Multidimensional Analysis of Empathetic Systems (2024.eacl-long)
Copied to clipboard
| Challenge: | Empathetic dialogue systems have received significant attention, but no systematic review has verified these limitations. |
| Approach: | They analyze 21 empathetic dialogue systems using automated methods to examine their progress. |
| Outcome: | The results show that empathetic dialogue systems lack specificity, reflection levels, diversity . the results also offer guidance for developing future systems . |
Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation (2024.acl-long)
Copied to clipboard
Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay-Yoon Lee, Mohit Iyyer, Andrew McCallum
| Challenge: | Low-resource tasks such as semi-supervised sequence generation require expert knowledge and cost. |
| Approach: | They propose a method for semi-supervised sequence generation where few examples are too scarce to fine tune a model. |
| Outcome: | The proposed method can generalize better than its teacher to unseen examples on semi-supervised sequence generation tasks. |
FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Datasets Dependency (2025.acl-srw)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) have emerged as a promising solution for decomposing large language model representations into interpretable features. |
| Approach: | They propose a method that trains SAEs on the model’s own synthetic dataset and a model-specific model to capture model-internal features. |
| Outcome: | The proposed method outperforms SAEs trained on web-based datasets and exhibits lower Fake Feature Ratio in 5 out of 7 models. |
Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models (2024.findings-naacl)
Copied to clipboard
Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, Andrew Lan
| Challenge: | Multiple-choice questions (MCQs) are easy to administer and grade . but crafting high-quality distractors remains labor-intensive and limited scalability . |
| Approach: | They propose to automate the generation of distractors in math MCQs by using large language models to generate distractors. |
| Outcome: | The proposed methods can generate valid distractors, but they are less adept at anticipating common errors or misconceptions among real students. |
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained large language models can be fine-tuned with instruction tuning to align the model responses with human intentions. |
| Approach: | They investigate the effectiveness of parameter efficient fine-tuning (PEFT) of the Q-Former with visual reasoning benchmarks ScienceQA and IconQA. |
| Outcome: | The proposed model achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. |
Simulated Students in Tutoring Dialogues: Substance or Illusion? (2026.acl-long)
Copied to clipboard
| Challenge: | evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. |
| Approach: | They propose to define the student simulation task and benchmark a wide range of student simulation methods on these metrics. |
| Outcome: | The proposed evaluation metrics show that prompting strategies perform poorly on a real-world tutoring dialogue dataset. |
Improved Latent Tree Induction with Distant Supervision via Span Constraints (2021.emnlp-main)
Copied to clipboard
Zhiyang Xu, Andrew Drozdov, Jay Yoon Lee, Tim O’Gorman, Subendhu Rongali, Dylan Finkbeiner, Shilpa Suresh, Mohit Iyyer, Andrew McCallum
| Challenge: | Distant supervision is not a practical way to perform unsupervised syntactic parsing. |
| Approach: | They propose a technique that uses distant supervision to improve unsupervised constituency parsing by using phrase bracketing. |
| Outcome: | The proposed method improves constituency parsing on English WSJ Penn Treebank by more than 5 F1 compared with full parse tree annotations. |
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models (2024.lrec-main)
Copied to clipboard
Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, Rada Mihalcea
| Challenge: | Recent advances in large language models have led to misleading public discourse that “it’s all been solved.” |
| Approach: | They identify 14 research areas encompassing 45 research directions that require new research and are not directly solvable by LLMs. |
| Outcome: | The research areas identified are 45 research directions that require new research and are not directly solvable by LLMs. |
From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate features disentangle concepts from activations of neural networks are limited by their quality . current methods for concept identification and steering are sparse autoencoders, but they are not reliable. |
| Approach: | They propose to evaluate how well featurization methods disentangle one concept from another . they use sentiment, domain, voice, and tense to steer these features . |
| Outcome: | The proposed evaluations show that featurization methods are insufficient to establish steering selectivity . the results suggest that steering a feature affects many concepts despite a near absence of interaction effects. |
Deduplicating Training Data Makes Language Models Better (2022.acl-long)
Copied to clipboard
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini
| Challenge: | Existing language modeling datasets contain near-duplicate examples and long repetitive substrings. |
| Approach: | They develop tools that allow us to deduplicate existing language modeling datasets . they found that over 1% of the unprompted output of language models is copied verbatim . |
| Outcome: | The proposed tools reduce train-test overlap, which affects over 4% of validation sets, and improve model accuracy. |