Papers by Yujie Lu
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing learning metrics are limited to tasks where large human ratings are available. |
| Approach: | They propose a model-based natural language generation (NLG) evaluation metric that is highly correlated with human judgements without requiring human annotation. |
| Outcome: | The proposed metric outperforms all prior unsupervised metrics on multiple NLG tasks including translation, image captioning, and WebNLG text generation. |
ULN: Towards Underspecified Vision-and-Language Navigation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-and-language navigation models are brittle to multi-level language underspecification. |
| Approach: | They propose to use multi-level underspecified instructions to guide agents . they propose to learn GSS for navigation agent to ground multi- level instructions . experimental results show existing VLN models are still brittle to multi-language underspecification . |
| Outcome: | Experimental results show that the proposed framework outperforms baselines on ULN by 10% relative success rate across all levels. |
Let’s Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thought (2023.emnlp-main)
Copied to clipboard
Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, William Wang
| Challenge: | Existing studies show vision-language systems can reason about images using natural language, but their capacity for video reasoning remains underexplored. |
| Approach: | They propose to frame video reasoning as the sequential understanding of a small number of keyframes, thereby leveraging the power and robustness of vision-language systems' capacity to reason about images using natural language. |
| Outcome: | The proposed models can generate multiple intermediate keyframes and predict future keyframe, and they perform poorly on GPT-4, GPT-3, and VICUNA. |
Continual Dialogue State Tracking via Reason-of-Select Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research on dialogue systems has focused on domain-specific offline systems lacking adaptation abilities. |
| Approach: | They propose a Reason-of-Select distillation method that enhances smaller models with a novel "meta-reasoning" capability. |
| Outcome: | Experiments show that the proposed method significantly improves the performance and generalization capabilities of existing models. |
Recurrent Knowledge Identification and Fusion for Language Model Continual Learning (2025.acl-long)
Copied to clipboard
Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S. Yu, Xu Chu, Xiao-Ming Wu
| Challenge: | Continual learning (CL) is crucial for large language models without costly retraining. |
| Approach: | They propose a framework for recurrent knowledge identification and fusion that enables dynamic estimation of parameter importance distributions to enhance knowledge transfer. |
| Outcome: | The proposed framework mitigates catastrophic forgetting and enhances knowledge transfer. |
ChildTalk: A Multi-Dialect Chinese Child Speech Corpus with Full-Length Child–Caregiver Conversations for Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Automatic speech recognition (ASR) for children remains challenging due to developmental variability and the scarcity of high-quality corpora. |
| Approach: | They propose a large-scale Chinese child speech corpus that contains 112.5 hours of speech from 498 children and 500 caregivers. |
| Outcome: | The proposed model improves in-domain and cross-domain performance on children's speech. |
Multimodal Procedural Planning via Dual Text-Image Prompting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Embodied agents have demonstrated performance in following instructions informed by texts and images . however, the potential of models providing useful guidelines for humans to complete tasks remains underexplored . |
| Approach: | They propose a multimodal procedural planning task that generates paired text-image plans . this task provides more complementary and informative guidance than unimodal plans a . authors propose modality prompting methods that leverage zero-shot reasoning ability . |
| Outcome: | The proposed method improves the interaction in dual modalities and provides more information than unimodal plans. |
Imagination-Augmented Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for Natural Language Understanding focus on textual signals, which hinders models from learning efficiently from limited data samples. |
| Approach: | They propose an Imagination-Augmented Cross-modal Encoder to solve natural language understanding tasks from a novel learning perspective. |
| Outcome: | The proposed learning paradigm bridges the gap between human and agent language understanding in both linguistic and perceptual procedures. |
Towards LLM-driven Dialogue State Tracking (2023.emnlp-main)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. |
| Approach: | They present a framework for a domain-slot instruction tuning method that allows LDST to achieve performance on par with ChatGPT. |
| Outcome: | The proposed framework performs better in zero-shot and few-shot settings than previous SOTA methods. |
Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for mental health support are shallow and heuristic, e.g., analyzing emotions and generating comforting responses. |
| Approach: | They propose to use cognitive distortion detection to perform diagnosis on the patient’s speech via three stages: subjectivity assessment to separate the facts and the thoughts; contrastive reasoning to elicit the reasoning processes supporting and contradicting the thoughts and schema analysis to summarize the cognition schemas. |
| Outcome: | The proposed system improves on ChatGPT for cognitive distortion detection while generating high-quality rationales approved by human experts. |
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter. |
| Approach: | They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I. |
| Outcome: | The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers . |
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing tools for text-to-image synthesis can visualize machine imaginations for a given context. |
| Approach: | They propose a framework that uses machine-generated images to guide language models in open-ended text generation. |
| Outcome: | The proposed framework is effective on open-ended text generation tasks while showing minor degeneration. |
Few-Shot Document-Level Event Argument Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Event argument extraction (EAE) has been well studied at the sentence level but under-explored at the document level. |
| Approach: | They propose a Few-Shot Document-Level Event Argument Extraction benchmark to capture event arguments that actually spread across sentences in documents. |
| Outcome: | The proposed task is very challenging with low performance and limited learning process . argument extraction depends on context from multiple sentences and learning process limited to very few examples . |
GeoEdit: Geometric Knowledge Editing for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Li-Ming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S. Yu, Xiao-Ming Wu
| Challenge: | Existing training-based model editing methods struggle to incorporate new knowledge while preserving unrelated general knowledge. |
| Approach: | They propose a framework that uses geometric relationships to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. |
| Outcome: | The proposed framework avoids updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model’s generalization ability. |
How Good Are LLMs at Out-of-Distribution Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is crucial for ensuring AI safety . large language models (LLMs) are becoming more prevalent due to their scale, pre-training objectives, and paradigms used for inference. |
| Approach: | They propose to use large language models to investigate out-of-distribution (OOD) detection in machine learning. |
| Outcome: | The proposed method outperforms other OOD detectors in zero-grad and fine-tuning scenarios. |
AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Jian Li, Xiaoyu Dong, Pengfei Xu, Xiaohui Zhou, Yujia Zhang, Zexin Lu, Yasha Wang, Alan Zhao, Xu Chu, Xiao-Ming Wu
| Challenge: | Recent model merging-based methods struggle to effectively manage the trade-off between learning new knowledge and preventing catastrophic forgetting. |
| Approach: | They propose a model merging framework that utilizes learning and forgetting signals from the training trajectory to dynamically monitor the model’s training status. |
| Outcome: | The proposed framework achieves significant performance improvements over existing state-of-the-art methods on three CL benchmarks with various model sizes (from 770M to 13B). |
Zero-shot Cross-domain Dialogue State Tracking via Context-aware Auto-prompting and Instruction-following Contrastive Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Previous studies have implemented slot-based input improvements, such as schema-driven descriptions and question-answering formats, but still suffer from negative transfer for seen slots and inefficient transfer for unseen slots due to the significant source-target domain gap. |
| Approach: | They propose a framework that generates dynamic, context-aware slot queries to improve model transferability by penalizing deviations from the provided instructions. |
| Outcome: | Experiments on two datasets show that the proposed model performs better than existing models on the restaurant domain. |