Papers by Lu Dai
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current task-oriented dialogue systems focus on multi-turn text/speech interaction, then call back-end APIs to perform task. |
| Approach: | They propose a GUI-based task-oriented dialogue system that can perform GUI operations on real APPs without invoking TOD-specific backend APIs. |
| Outcome: | The proposed GUI-based task-oriented dialogue system can perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. |
Learning In-context Learning for Named Entity Recognition (2023.acl-long)
Copied to clipboard
Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, Le Sun
| Challenge: | Existing methods to recognize entities in text are limited by the diversity of entity types and the lack of high-quality annotations. |
| Approach: | They propose an in-context learning-based NER approach that can inject in-const NER ability into PLMs and recognize entities of novel types on-the-fly using only a few demonstrative instances. |
| Outcome: | The proposed method outperforms the PLMs+fine-tuning counterparts on 4 few-shot NER datasets and significantly outperformed the Plms+initialized extractors. |
mCLIP: Multilingual CLIP via Cross-lingual Transfer (2023.acl-long)
Copied to clipboard
Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, Wenping Wang
| Challenge: | Existing multilingual vision-language pretrained models are biased towards English due to the lack of sufficient non-English image-text pairs. |
| Approach: | They propose to train a retrieval-efficient dual-stream multilingual VLP model by aligning CLIP model and a multilingual text encoder through a novel Triangle Cross-modal Knowledge Distillation method. |
| Outcome: | Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval tasks. |
Unified Structure Generation for Universal Information Extraction (2022.acl-long)
Copied to clipboard
| Challenge: | Information extraction suffers from its varying targets, heterogeneous structures, and demand-specific schemas. |
| Approach: | They propose a unified text-to-structure generation framework, namely UIE, which can universally model different IE tasks, adaptively generate targeted structures, and collaboratively learn general IE abilities from different knowledge sources. |
| Outcome: | The proposed framework can model different IE tasks, generate targeted structures, and learn general IE abilities from different knowledge sources. |
CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on narrow tasks such as multiple-choice cloze tests, isolated translation, or simple paraphrasing. |
| Approach: | They propose a benchmark to measure Chinese idioms' cultural and contextual nuances . they evaluate 2,937 human-verified examples covering 1,765 common idiomes . |
| Outcome: | The proposed benchmarks achieve 95% accuracy on Evaluative Connotation, but only 85% on Appropriateness and 40% top-1 accuracy in Open Cloze. |
Improve Dense Passage Retrieval with Entailment Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for dense retrieval have demonstrated remarkable performance in IR tasks. |
| Approach: | They propose a method to improve the embedding of dense retrievers by using existence claim as a bridge. |
| Outcome: | The proposed method can be plugged into current dense retrieval methods and the results are published in the journal Nature. |
RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world Scenarios (2025.findings-emnlp)
Copied to clipboard
Fei Zhao, Chengqiang Lu, Yufan Shen, Qimeng Wang, Yicheng Qian, Haoxin Zhang, Yan Gao, null Yiwu, Yao Hu, Zhen Wu, Shangyu Xing, Xinyu Dai
| Challenge: | RealBench is the first Chinese multimodal multi-image dataset . the dataset contains 9393 samples and 69910 images . |
| Approach: | They propose to create a Chinese multimodal multi-image dataset using 21 models . they use closed-source models that support multi-inputs as well as open-source visual and video models a . |
| Outcome: | The first Chinese multimodal multi-image dataset contains 9393 samples and 69910 images. |
MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Correction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown growing potential in molecular sciences, but they often produce chemically inaccurate descriptions and struggle to recognize or justify potential errors. |
| Approach: | They propose a benchmark to assess LLMs on error detection and correction in molecular descriptions. |
| Outcome: | The proposed benchmark targets LLMs on error detection and correction in molecular descriptions. |
CARD: Cross-modal Agent Framework for Generative and Editable Residential Design (2025.emnlp-main)
Copied to clipboard
| Challenge: | Architectural design automation has made significant progress, but the complexity of open-world environments makes residential design a challenging task. |
| Approach: | They propose a framework that leverages a system of specialized cross-modal agents to adapt to open-world residential design. |
| Outcome: | The proposed framework enables users to generate and edit residential design without requiring specialized expertise. |
Encoding and Fusing Semantic Connection and Linguistic Evidence for Implicit Discourse Relation Recognition (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies use one attention mechanism to improve contextual semantic representation learning for implicit discourse relation recognition (IDRR). |
| Approach: | They propose a Multi-Attentive Neural Fusion model to fuse linguistic evidence and semantic connection for IDRR by using a Dual Attention Network and an Offset Matrix Network. |
| Outcome: | The proposed model achieves state-of-the-art on the PDTB 3.0 corpus. |
FloorPlan-LLaMa: Aligning Architects’ Feedback and Domain Knowledge in Architectural Floor Plan Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for floor plan generation rely on statistical metrics like FID, GED, and PSNR, which fail to evaluate using domain knowledge. |
| Approach: | They propose to use a first floor plan dataset to train a floor plan generation model based on a multi-dimensional preference score and a textual analysis to integrate architects’ professional expertise and preferences. |
| Outcome: | The proposed model outperforms baseline models in text-conditional and class-condition tasks and is more rational and aligns better with human preferences. |
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied Instruction Following has shown an impressive success rate when the environment has been seen in training, but when deployed in an unseen environment, it tends to struggle when deployed with an unsightly environment. |
| Approach: | They propose to explicitly align the agent’s hidden states with the instructions via contrastive learning to bridge the semantic gap between high-level language instructions and the agent's low-level action space. |
| Outcome: | The proposed meta-actions achieve a 4.5% success rate in unseen environments compared to a strong multi-modal Transformer baseline . |
Deploying Multi-task Online Server with Large Language Model (2025.coling-industry)
Copied to clipboard
| Challenge: | In the industry, numerous natural language processing tasks are deployed online . traditional approaches tackle each task separately by its own network and pipeline . |
| Approach: | They propose a three-stage multi-task learning framework for large language models . it involves task filtering, fine-tuning on high-resource tasks, and finally fine- tuning on all tasks . |
| Outcome: | The proposed framework reduces up to 90% of overhead while reducing latency and resource usage. |
DUET: Joint Exploration of User–Item Profiles in Recommendation System (2026.findings-acl)
Copied to clipboard
Yue Chen, Yifei Sun, Lu Wang, Fangkai Yang, Pu Zhao, Minjie Hong, Yifei Dong, Minghua He, Nan Hu, Jianjin Zhang, Zhiwei Dai, Yuefeng Zhan, Weihao Han, Hao Sun, Qingwei Lin, Weiwei Deng, Feng Sun, Qi Zhang, Saravan Rajmohan, Dongmei Zhang
| Challenge: | Existing LLMs are opaque and difficult to interpret, resulting in limited interpretability. |
| Approach: | They propose an interaction-aware profile generator that jointly produces user and item profiles conditioned on both user history and item evidence. |
| Outcome: | The proposed model outperforms baselines on three real-world datasets. |
ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning (2025.emnlp-demos)
Copied to clipboard
Yichen Lu, Wei Dai, Jiaen Liu, Ching Wing Kwok, Zongheng Wu, Xudong Xiao, Ao Sun, Sheng Fu, Jianyuan Zhan, Yian Wang, Takatomo Saito, Sicheng Lai
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in Machine Translation (MT) tasks. |
| Approach: | They propose a translation agent system designed for multimodal input that leverages visual and contextual background information to enhance the translation process. |
| Outcome: | The proposed translation agent achieves significantly higher translation quality in subtitle generation and general translation tasks compared to previous state-of-the-art systems. |
Large Dual Encoders Are Generalizable Retrievers (2022.emnlp-main)
Copied to clipboard
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, Yinfei Yang
| Challenge: | Experimental results show that dual encoders outperform sparse and dense retrievers on the BEIR dataset significantly. |
| Approach: | They challenge belief that bottleneck layer is too limited for out-of-domain generalization . they scale up the model while keeping bottleneck as a single dot-product with a fixed size . |
| Outcome: | The proposed model outperforms sparse and dense retrievers on the BEIR dataset significantly. |
Efficient Inference for Large Language Models –Algorithm, Model, and System (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | Inference of LLMs incurs high computational costs, memory access overhead, and memory usage, leading to inefficiencies in terms of latency, throughput, power consumption, and storage. |
| Approach: | This tutorial introduces the basics of efficient inference for LLMs and explains how to diagnose efficiency bottlenecks for a given workload on specific hardware. |
| Outcome: | The tutorial introduces the basic concepts of modern LLMs, software and hardware. |
WASA: WAtermark-based Source Attribution for Large Language Model-Generated Data (2025.findings-acl)
Copied to clipboard
Xinyang Lu, Jingtan Wang, Zitong Zhao, Zhongxiang Dai, Chuan-Sheng Foo, See-Kiong Ng, Bryan Kian Hsiang Low
| Challenge: | Large language models (LLMs) have impressive performance but intellectual property concerns are looming . a framework that can be used to perform source attribution for LLMs can be developed. |
| Approach: | They propose a framework that enables an LLM to generate synthetic texts with embedded watermarks that contain information about their source. |
| Outcome: | The proposed framework achieves source attribution accuracy and robustness against adversaries. |
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent large-scale vision-language pre-training models are powerful in multimodal classification and retrieval tasks. |
| Approach: | They propose to augment a vision-language pre-training model with a textual pre-trained language model . the model achieves 44.5% zero-shot accuracy on multimodal generation tasks . |
| Outcome: | The proposed model achieves 44.5% zero-shot accuracy on open-ended visual question answering and image captioning tasks. |
Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt-tuning methods for event argument extraction lack entity information . eAE is a key step of event extraction, but it requires a pre-trained language model to extract event arguments. |
| Approach: | They propose a prompt-tuning method that takes advantage of entity information and pre-trained language models. |
| Outcome: | The proposed method outperforms the state-of-the-art prompt-tuning methods on an english dataset. |
Towards Interpretable Clinical Diagnosis with Bayesian Network Ensembles Stacked on Entity-Aware CNNs (2020.acl-main)
Copied to clipboard
| Challenge: | a novel framework for text-based diagnosis of diseases requires appropriate balance between accuracy and interpretability. |
| Approach: | They propose a framework that stacks Bayesian Network Ensembles on top of CNN to build an accurate yet interpretable diagnosis system. |
| Outcome: | The proposed framework outperforms the previous automatic diagnosis methods in accuracy performance and the diagnosis explanation of the framework is reasonable. |
D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing human-machine dialogue systems are not able to provide diagnostic information for depression diagnosis due to stigma associated with mental illness. |
| Approach: | They propose to construct a Chinese Dialogue Dataset for depression-diagnosis-oriented chat based on clinical depression diagnostic criteria. |
| Outcome: | The proposed system can be used to diagnose depression using a Chinese Dialogue Dataset. |
ConnPrompt: Connective-cloze Prompt Learning for Implicit Discourse Relation Recognition (2022.coling-1)
Copied to clipboard
| Challenge: | Existing paradigms for Implicit Discourse Relation Recognition (IDRR) do not exploit linguistic evidence embedded in the pre-training process. |
| Approach: | They propose a new paradigm to detect and classify relation sense between two text segments without an explicit connective. |
| Outcome: | The proposed method significantly outperforms the state-of-the-art algorithms even with fewer training data. |
Stealing Training Data from Large Language Models in Decentralized Training through Activation Inversion Attack (2025.acl-long)
Copied to clipboard
| Challenge: | Decentralized training is a resource-efficient framework to democratize training of large language models. |
| Approach: | They propose an activation inversion attack to exploit privacy leakage from training data . they construct a shadow dataset comprising text labels and corresponding activations . |
| Outcome: | The proposed attack surface is based on a shadow dataset and public datasets . the proposed attack model reconstructs training data from activations in victim decentralized training. |
Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing routing attacks depend on white-box access or heuristic prompts, rendering them ineffective in real-world black-box scenarios. |
| Approach: | They propose a cost-aware routing strategy that routes queries to the least-cost model . they propose heuristic prompts that are ineffective in real-world black-box scenarios . |
| Outcome: | The proposed approach significantly increases the routing rate to expensive models on queries of different distributions. |