Papers by Xiaodong Liu

Composable Text Controls in Latent Space with ODEs (2023.emnlp-main)

Challenge:	Existing datasets for human-like dialogue tasks are deficient due to the complexity of human conversations.
Approach:	They construct a large-scale Chinese E-commerce conversation corpus with 1 million dialogues, 20 million utterances, and 150 million words.
Outcome:	The proposed dataset includes 1 million multi-turn dialogues, 20 million utterances, and 150 million words.

Copied to clipboard

Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, Zhiting Hu

Green & Sustainable NLP

Challenge:	Existing approaches to composable text operations often require plug-and-play . a single LM can perform arbitrary text operation composition in the latent space .
Approach:	They propose an efficient approach for composable text operations in the latent space of text . they connect pretrained LMs to the laten space and adapt them to the space .
Outcome:	The proposed approach improves on existing methods in the latent space of text.

Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)

Copied to clipboard

Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Hongning Wang

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023.findings-emnlp)

Challenge:	Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data.
Approach:	They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain.
Outcome:	The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning.

Copied to clipboard

Yu Zhang, Hao Cheng, Zhihong Shen, Xiaodong Liu, Ye-Yi Wang, Jianfeng Gao

Self-Taught Agentic Long Context Understanding (2025.acl-long)

Challenge:	Pre-trained language models (LMs) have shown effectiveness in literature understanding tasks, especially when tuned via contrastive learning.
Approach:	They propose a multi-task contrastive learning framework that enables common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other.
Outcome:	The proposed framework outperforms state-of-the-art pre-trained language models on a comprehensive dataset.

Copied to clipboard

Yufan Zhuang, Xiaodong Yu, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Jingbo Shang, Zicheng Liu, Emad Barsoum

Information Retrieval

Challenge:	Extensive experiments across seven long-context tasks demonstrate that AgenticLU significantly outperforms state-of-the-art prompting methods and specialized long-consumer LLMs.
Approach:	They propose a framework to enhance an LLM's understanding of long-context questions by integrating targeted self-clarification with contextual grounding within an agentic workflow.
Outcome:	The proposed framework outperforms state-of-the-art prompting methods and specialized long-context LLMs in seven long-constitut tasks.

Understanding the Difficulty of Training Transformers (2020.emnlp-main)

Copied to clipboard

Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Jiawei Han

Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning (2025.coling-main)

Challenge:	Admin (Adaptive model initialization) is more stable, converges faster, and leads to better performance.
Approach:	They propose a model initialization algorithm to stabilize early training and unleash its full potential in the late stage.
Outcome:	The proposed model initialization method stabilizes early training and unleashes full potential in late stage.

Copied to clipboard

Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, Jianfeng Gao

Explainability & Interpretability in NLP

Challenge:	Large language models generate convincing, fluent explanations, but they often generate inconsistent explanations on different inputs.
Approach:	They propose a method that adapts large language models to generate more consistent explanations on related examples.
Outcome:	The proposed method yields a 10.0% relative explanation consistency improvement across a variety of question-answering datasets and generalizes to 7 out-of-distribution datasets not seen during finetuning (+4.5%)

Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach (2021.emnlp-main)

Copied to clipboard

Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao

AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation (2023.findings-acl)

Challenge:	Existing approaches to adversarial regularization treat adversarials and defending players equally, which is undesirable because only the defending player contributes to the generalization performance.
Approach:	They propose a method which formulates adversarial regularization as a Stackelberg game and induces a competition between a leader and a follower.
Outcome:	The proposed method outperforms existing adversarial regularization baselines on a set of machine translation and natural language understanding tasks.

Copied to clipboard

Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sebastien Bubeck, Jianfeng Gao

The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)

Challenge:	Existing MoE designs do not consider computational constraints (e.g., FLOPs, latency) Existing works in MoE consider homogeneous design where the same number of experts of the same size are placed uniformly throughout the network.
Approach:	They propose a framework for designing heterogeneous MoEs under computational constraints.
Outcome:	The proposed framework achieves 4x inference speedup and FLOPs reduction over manual models and within 1 BLEU point of MoE SwitchTransformer over benchmark datasets for NMT.

Copied to clipboard

Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao

Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading (P19-1)

Challenge:	MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models .
Approach:	They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop.
Outcome:	The proposed model can significantly compress a large model without significant performance drop.

Copied to clipboard

Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, Jianfeng Gao

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (2024.findings-naacl)

Challenge:	a new approach to contentful neural conversation is proposed . end-to-end models are effective in learning fluent responses, but their responses are often vacuous and uninformative.
Approach:	They propose a model that provides the conversation model with relevant text on the fly as a source of external knowledge.
Outcome:	The proposed model improves the informativeness and diversity of generated output compared to previous methods.

Copied to clipboard

Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, Jianfeng Gao

E-ConvRec: A Large-Scale Conversational Recommendation Dataset for E-Commerce Customer Service (2022.lrec-1)

Challenge:	Existing static benchmarks do not guarantee that models can use the provided evidence for answering, which is essential to avoid hallucination when the required knowledge is new or private.
Approach:	They propose to automatically perturb existing static one for dynamic evaluation by using a chatGPT framework and a set of open-domain QA datasets.
Outcome:	The proposed framework generates new test cases on two open-domain QA datasets and is human-readable and useful to trigger hallucination in LLMs.

Copied to clipboard

Meihuizi Jia, Ruixue Liu, Peiying Wang, Yang Song, Zexi Xi, Haobin Li, Xin Shen, Meng Chen, Jinhui Pang, Xiaodong He

M2PA: A Multi-Memory Planning Agent for Open Worlds Inspired by Cognitive Theory (2025.findings-acl)

Challenge:	Recent research has focused on developing conversational recommendation system (CRS), which provides valuable recommendations to users through conversations.
Approach:	They construct an authentic Chinese dialogue dataset consisting of over 25k dialogues and 770k utterances, which contains user profile, product knowledge base, and multiple sequential real conversations between users and recommenders.
Outcome:	The proposed dataset contains user profile, product knowledge base, and multiple sequential real conversations between users and recommenders.

Copied to clipboard

YanfangZhou YanfangZhou, Xiaodong Li, Yuntao Liu, Yongqiang Zhao, Xintong Wang, Zhenyu Li, Jinlong Tian, Xinhai Xu

Linguistics & Cognitive NLP

Challenge:	Open-world planning poses a challenge due to complex environments and task diversity . recent work shows that large language models (LLMs) lack the ability to connect to agents' experiences .
Approach:	They propose an open-world multi-memory planning agent that combines large language models with human-like multi-mesh systems to leverage their strengths.
Outcome:	The proposed agent outperforms state-of-the-art agents on 50 Minecraft tasks in zero-shot learning.

Agent Laboratory: Using LLM Agents as Research Assistants (2025.findings-emnlp)

Copied to clipboard

Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, Emad Barsoum

Programming Languages in NLP

Challenge:	Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process.
Approach:	Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process.
Outcome:	Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process.

Stabilizing Efficient Reasoning with Step-Level Advantage Selection (2026.findings-acl)

Copied to clipboard

Han Wang, Xiaodong Yu, Jialian Wu, Jiang Liu, Ximeng Sun, Mohit Bansal, Zicheng Liu

Green & Sustainable NLP

Challenge:	Large language models generate long and verbose reasoning traces at inference time . short context post-training alone induces substantial reasoning compression .
Approach:	They propose a step-level advantage selection approach that reduces reasoning length by over 30% . they propose to use GRPO without any length-aware objective to train models in a shorter context window .
Outcome:	The proposed approach reduces average reasoning length by over 30% while improving Pass@1 accuracy by 3.79 points over the strongest length-aware baseline.

Token-wise Curriculum Learning for Neural Machine Translation (2021.findings-emnlp)

Copied to clipboard

Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Tuo Zhao

Machine Translation

Challenge:	Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of “easy” samples from training data at the early stage of training.
Approach:	They propose a token-wise curriculum learning approach that creates sufficient amounts of easy samples from training data.
Outcome:	The proposed approach outperforms baselines on five language pairs on low-resource languages.

Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)

Copied to clipboard

Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen

Reliable Use of Lemmas via Eligibility Reasoning and Section-Aware Reinforcement Learning (2026.acl-short)

Challenge:	'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size .
Approach:	They propose to train a subnetwork of 'lottery tickets' to match the full model's performance.
Outcome:	The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large .

Copied to clipboard

Zhikun Xu, Xiaodong Yu, Ben Zhou, Jiang Liu, Jialian Wu, Ze Wang, Ximeng Sun, Hao Chen, Zicheng Liu

Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities (2025.naacl-long)

Challenge:	Recent large language models (LLMs) perform strongly on mathematical benchmarks but often import conclusions without validating assumptions.
Approach:	They propose a model that encodes a lemma specification and trains with reinforcement learning and section-aware loss masking to assign penalty to the section responsible for errors.
Outcome:	The proposed model performs well on benchmarks but often misapplyes lemmas . the model is able to encode the specification and train with reinforcement learning .

Copied to clipboard

Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, Jianfeng Gao

K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce (2021.findings-emnlp)

Challenge:	Recent research shows that Large Language Models (LLMs) are vulnerable to automated jailbreak attacks.
Approach:	They propose a framework that crafts adversarial LLMs with enhanced jailbreak ability.
Outcome:	ADV-LLM significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs.

Copied to clipboard

Song Xu, Haoran Li, Peng Yuan, Yujia Wang, Youzheng Wu, Xiaodong He, Ying Liu, Bowen Zhou

Stochastic Answer Networks for Machine Reading Comprehension (P18-1)

Challenge:	Existing pre-trained language models are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios.
Approach:	They propose a knowledge-injected pre-trained language model that can be transferred to both natural language understanding and generation tasks.
Outcome:	The proposed model significantly outperforms baselines across the board in e-commerce scenarios.

Copied to clipboard

Xiaodong Liu, Yelong Shen, Kevin Duh, Jianfeng Gao

Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering (2023.acl-short)

Challenge:	Several recent MRC models employ multi-step reasoning . we show that the use of a stochastic prediction dropout improves robustness .
Approach:	They propose a stochastic answer network that simulates multi-step reasoning in machine reading comprehension.
Outcome:	The proposed model improves robustness and results competitive with state-of-the-art models on the Stanford Question Answering Dataset and Microsoft MAchine Reading COmprehension Dataset.

Copied to clipboard

Hao Cheng, Hao Fang, Xiaodong Liu, Jianfeng Gao

SOTOPIA-: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents (2025.acl-long)

Challenge:	Existing dense retrieval models are parameter-inefficient and underperform sparse counterparts.
Approach:	They propose a task-aware specialization for dEnse Retrieval architecture that enables parameter sharing by interleaving shared and specialized blocks in a single encoder.
Outcome:	The proposed architecture surpasses BM25 on questions and passages using 60% of the parameters as bi-encoder dense retrievers.

Copied to clipboard

Wenyuan Zhang, Tianyun Liu, Mengxiao Song, Xiaodong Li, Tingwen Liu

Challenge:	Existing studies on the social simulation of large language model intelligent agents have shown that even expert agents 1 perform significantly worse on challenging social tasks compared to expert agents.
Approach:	They propose a framework that dynamically injects a variety of social strategies into expert agents, thereby automating the construction of high-quality social dialogue training corpus.
Outcome:	The proposed framework enables the integration of social strategies into language agents and improves their performance on social tasks.

Learning to Decouple Relations: Few-Shot Relation Classification with Entity-Guided Attention and Confusion-Aware Training (2020.coling-main)

Copied to clipboard

Yingyao Wang, Junwei Bao, Guangyi Liu, Youzheng Wu, Xiaodong He, Bowen Zhou, Tiejun Zhao

Information Extraction & Text Mining

Challenge:	Existing few-shot relation classifiers struggle to distinguish them with few annotated instances due to high co-occurrence of some relations .
Approach:	They propose a few-shot relation classification model with two mechanisms to decouple easily-confused relations.
Outcome:	The proposed model achieves comparable and even better results to strong baselines in terms of accuracy.

ARCH: Efficient Adversarial Regularized Training with Caching (2021.findings-emnlp)

Copied to clipboard

Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao

MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization (2024.findings-acl)

Challenge:	Existing approaches to regularize models require generating a perturbation for each sample in each epoch.
Approach:	They propose an adversarial regularization method where perturbations are generated and cached once every several epochs.
Outcome:	The proposed method significantly eases the computational burden (saves up to 70% of computational time) it produces a notably better (in most of the tasks) or comparable model generalization.

Copied to clipboard

Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, Maosong Sun

Multimodality

Challenge:	Scientific data visualization is an essential process in research, but its use of large language models remains unexplored.
Approach:	They propose a model-agnostic LLM agent framework to automate scientific data visualization tasks.
Outcome:	The proposed framework improves performance of commercial and open-source models.

Unsupervised Deep Structured Semantic Models for Commonsense Reasoning (N19-1)

Copied to clipboard

Shuohang Wang, Sheng Zhang, Yelong Shen, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Jing Jiang

Knowledge-Rich Self-Supervision for Biomedical Entity Linking (2022.findings-emnlp)

Challenge:	Existing methods for commonsense reasoning rely on human-crafted features and knowledge bases, but unsupervised learning is not feasible due to the lack of labeled training data or comprehensive knowledge bases.
Approach:	They propose two unsupervised models based on the Deep Structured Semantic Models framework to tackle two commonsense reasoning tasks: Winograd Schema Challenge (WSC) and Pronoun Disambiguation (PDP).
Outcome:	The proposed models capture contextual information in the sentence and co-reference information between pronouns and nouns, and achieve significant improvement over previous state-of-the-art approaches.

Copied to clipboard

Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon

HittER: Hierarchical Transformers for Knowledge Graph Embeddings (2021.emnlp-main)

Challenge:	Entity linking is challenging in high-value domains with myriad entities . standard classification approaches suffer from the annotation bottleneck .
Approach:	They propose a self-supervised approach to learn domain knowledge for biomedical entity linking . it generates self-reported mention examples on unlabeled text and trains contextual encoder .
Outcome:	The proposed method outperforms existing methods by 20 points in accuracy on biomedical datasets.

Copied to clipboard

Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji

Few-Shot Table Understanding: A Benchmark Dataset and Pre-Training Baseline (2022.coling-1)

Challenge:	Existing knowledge graph embedding methods to learn representations of knowledge graphs are conceptually simple and can be applied to tasks like factoid question answering (Saxena et al., 2020) and reasoning.
Approach:	They propose a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity’s neighborhood.
Outcome:	The proposed model achieves state-of-the-art on multiple link prediction datasets and can be integrated into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.

Copied to clipboard

Ruixue Liu, Shaozu Yuan, Aijun Dai, Lei Shen, Tiangang Zhu, Meng Chen, Xiaodong He

Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension (N19-1)

Challenge:	Pre-trained language models have demonstrated their effectiveness for few-shot table understanding, but few-shoot table understanding is rarely explored due to the deficiency of public table pre-training corpus and well-defined downstream benchmark tasks.
Approach:	They establish a benchmark dataset and use it to explore few-shot table understanding in Chinese.
Outcome:	The proposed model improves the few-shot table understanding in Chinese.

Copied to clipboard

Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, Jianfeng Gao

TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games (2025.emnlp-main)

Challenge:	Existing models for Machine Reading Comprehension (MRC) are small, compared to their size, and there are many studies on using pre-trained word embeddings and back-translation approaches to improve model generalization.
Approach:	They propose a multi-task learning framework to learn a machine reading comprehension model that can be applied to a wide range of MRC tasks in different domains.
Outcome:	The proposed model can be applied to a wide range of MRC tasks in different domains.

Copied to clipboard

Prakamya Mishra, Jiang Liu, Jialian Wu, Xiaodong Yu, Zicheng Liu, Emad Barsoum

Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation (2022.naacl-main)

Challenge:	Recent advances in large reasoning models (LRMs) have driven significant breakthroughs across various reasoning tasks including deductive, arithmetic, commonsense, relational, and symbolic reasoning.
Approach:	They propose a programmatic approach to evaluate basic strategic, spatial, and logical reasoning abilities in large reasoning models through four two-player Tic-Tac-Toe-style games that humans can effortlessly solve from a young age.
Outcome:	The proposed model performs 41% lower on TTT-Bench than MATH 500 and AIME 2024 models, while the larger models perform better on longer reasoning traces.

Copied to clipboard

Guangyi Liu, Zichao Yang, Tianhua Tao, Xiaodan Liang, Junwei Bao, Zhen Li, Xiaodong He, Shuguang Cui, Zhiting Hu

Text Generation

Challenge:	Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss.
Approach:	They propose an Edit-Invariant Sequence Loss method which computes the matching loss of a target sequence with all n-grams in the generated sequence.
Outcome:	The proposed method outperforms the common CE loss and strong baselines on a wide range of tasks.

Chain-of-Skills: A Configurable Model for Open-Domain Question Answering (2023.acl-long)

Copied to clipboard

Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, Jianfeng Gao

Information Retrieval

Challenge:	Using customized retrieval models, model transferability and scalability are limited.
Approach:	They propose a modular retrieval model where individual modules correspond to key skills that can be reused across datasets.
Outcome:	The proposed model outperforms self-supervised retrievers in zero-shot evaluations and achieves state-of-the-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.

UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021.acl-long)

Copied to clipboard

Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge (2022.findings-emnlp)

Challenge:	Recent work on open-domain question answering focuses on either extractive or generative readers exclusively.
Approach:	They propose a hybrid approach to extractive and generative readers that leverages both models.
Outcome:	The proposed approach outperforms state-of-the-art models on NaturalQuestions and TriviaQA respectively.

Copied to clipboard

Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao

Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers (2023.acl-long)

Challenge:	Existing open-domain question answering methods rely on the retriever to gather all evidence in isolation, but our approach uses an intermediary module to perform a chain of reasoning over the retrieved set.
Approach:	They propose a new open-domain question answering framework that integrates an intermediary module into the current retriever-reader pipeline and integrates it into the model.
Outcome:	The proposed framework outperforms the state-of-the-art on two OTT-QA datasets with an exact match score of 47.3 (45% relative gain).

Copied to clipboard

Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)

Challenge:	Recent work in NLP has shown that pretrained language models have made notable progress toward generalization to unseen tasks.
Approach:	They propose to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise.
Outcome:	The proposed model outperforms similar-sized baseline models on prompted NLP benchmarks and rivals the state-of-the-art model with only 8% of its parameters.

Copied to clipboard

Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou

Multilinguality

Challenge:	XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios .
Approach:	They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora.
Outcome:	The proposed dataset is labeled in English and includes only natural language understanding tasks.

Generation-Augmented Retrieval for Open-Domain Question Answering (2021.acl-long)

Copied to clipboard

Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Weizhu Chen

RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2020.acl-main)

Challenge:	Existing approaches to answer open-domain questions use sparse representations and sparsity.
Approach:	They propose a method which augments a query by generating relevant contexts from heuristically discovered contexts without external supervision.
Outcome:	The proposed approach outperforms state-of-the-art dense retrieval methods on natural questions and triviaQA datasets.

Copied to clipboard

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, Matthew Richardson

Programming Languages in NLP

Challenge:	Existing semantic parsing models struggle to generalize to unseen database schemas.
Approach:	They propose a framework to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder.
Outcome:	The proposed framework boosts the match accuracy to 57.2% on the spider dataset, surpassing its best counterparts by 8.7%.

DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning (2026.findings-acl)

Copied to clipboard

Chao Huang, Zeliang Zhang, Jiang Liu, Ximeng Sun, Jialian Wu, Xiaodong Yu, Ze Wang, Chenliang Xu, Emad Barsoum, Zicheng Liu

A Hybrid Neural Network Model for Commonsense Reasoning (D19-60)

Challenge:	Multimodal large language models (MLLMs) have made rapid progress in perception and alignment, but their reasoning ability often lags behind strong text-only LLMs.
Approach:	They propose a method that transfers reasoning knowledge in the gradient space while preserving multimodal alignment.
Outcome:	Experiments on multimodal reasoning benchmarks show that DRIFT outperforms naive merging and standard SFT.

Copied to clipboard

Pengcheng He, Xiaodong Liu, Weizhu Chen, Jianfeng Gao

Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)

Challenge:	a hybrid neural network (HNN) model for commonsense reasoning is proposed . it combines language models and semantic similarity models to achieve new state-of-the-art results .
Approach:	They propose a hybrid neural network model for commonsense reasoning . it combines a masked language model and a semantic similarity model .
Outcome:	The proposed model outperforms the WNLI, WSC and PDP60 benchmarks on three commonsense reasoning tasks.

Copied to clipboard

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

Posterior Differential Regularization with f-divergence for Improving Model Robustness (2021.naacl-main)

Challenge:	Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training.
Approach:	They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model.
Outcome:	The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement)

Copied to clipboard

Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, Jianfeng Gao

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)

Challenge:	Recent studies show that pre-trained models suffer catastrophic degradation in out-of-domain generalization to datasets with domain shift or adversarial scenarios.
Approach:	They propose to regularize the posterior difference between clean and noisy inputs by using a Jacobian regularization framework and a virtual adversarial training framework.
Outcome:	The proposed framework can improve model robustness in fully supervised and semi-supervised settings.

Copied to clipboard

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao

NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation (2023.acl-long)

Challenge:	Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task.
Approach:	They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning.
Outcome:	The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder.

Copied to clipboard

Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Ming Gong, Lijuan Wang, Zicheng Liu, Houqiang Li, Nan Duan

Multimodality

Challenge:	Existing work generates long videos segment by segment sequentially, which is inefficient.
Approach:	They propose a Diffusion over Difference architecture for eXtremely Long video generation.
Outcome:	The proposed architecture reduces the average inference time from 7.55min to 26s (94.26%) and generates high-quality long videos with both global and local coherence.

Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience (2025.emnlp-main)

Copied to clipboard

Xi Wang, Songlei Jian, Shasha Li, Xiaopeng Li, Bin Ji, Ma Jun, Xiaodong Liu, Jing Wang, Jianfeng Zhang, Jie Yu, Feilong Bao, null Wangbaosheng

LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios (2025.acl-long)

Challenge:	Existing methods to generate human-aligned content with a “jailbreak prompt” are inefficient and repetitive, causing inefficiency and a lack of experience.
Approach:	They propose a framework that integrates past attack experiences to aid current jailbreak attempts.
Outcome:	The proposed framework improves both attack effectiveness and efficiency compared to the current black-box jailbreak method.

Copied to clipboard

Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu, Wei Zhang

Open Domain Question Answering with A Unified Knowledge Interface (2022.acl-long)

Challenge:	Existing benchmarks rarely focus on instruction-following in long-context scenarios or stability on different inputs.
Approach:	They propose a scalable dataset to evaluate LLMs’ instruction-following capabilities and stability across long contexts.
Outcome:	The proposed method evaluates LLMs’ instruction-following capabilities and stability across long contexts.

Copied to clipboard

Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, Jianfeng Gao

Structured Data in NLP

Challenge:	a retriever-reader framework is popular for open domain question answering . however, accessing heterogeneous knowledge sources through a unified interface remains unknown .
Approach:	They propose a retriever-reader framework that uses explicit knowledge to access heterogeneous knowledge sources through a unified interface.
Outcome:	The proposed framework can benefit from the expanded knowledge index, the authors show . their approach sets the single-model state-of-the-art on Natural Questions .

Metagent-P: A Neuro-Symbolic Planning Agent with Metacognition for Open Worlds (2025.findings-acl)

Copied to clipboard

YanfangZhou YanfangZhou, Yuntao Liu, Xiaodong Li, Yongqiang Zhao, Xintong Wang, Jinlong Tian, Zhenyu Li, Xinhai Xu

Beyond the Answer: Advancing Multi-Hop QA with Fine-Grained Graph Reasoning and Evaluation (2025.acl-long)

Challenge:	Recent advances in large language models (LLMs) show promising potential through their world knowledge and language processing capabilities in open-world planning.
Approach:	They propose a framework that integrates the world knowledge of large language models, symbolic reasoning capabilities of cognitive architectures, and metacognition to improve experience utilization.
Outcome:	The proposed framework outperforms current state-of-the-art methods in Minecraft and reduces the average replanning counts by 34% and exceeds the human success rate by 18.96%.

Copied to clipboard

Qichuan Liu, Chentao Zhang, Chenfeng Zheng, Guosheng Hu, Xiaodong Li, Zhihong Zhang

Structured Data in NLP

Challenge:	Existing evaluations of multi-hop question answering systems focus on comparing final answers of reasoning method and given ground-truths.
Approach:	They propose a "Planner-Executor-Reasoner" architecture that evaluates reasoning . they propose PER-DP and PER QA architectures that provide ground-truths .
Outcome:	The proposed model improves the performance of multi-hop question answering systems.

Targeted Adversarial Training for Natural Language Understanding (2021.naacl-main)

Copied to clipboard

Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi

LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models (2025.acl-long)

Challenge:	Existing adversarial training approaches focus on making adversarials less expensive or regularizing rather than replacing the standard training objective.
Approach:	They propose an algorithm to introspect current mistakes and prioritize adversarial training steps to where the model errs the most.
Outcome:	The proposed algorithm improves adversarial training for natural language understanding by introspecting mistakes and prioritizing training steps to where the model errs the most.

Copied to clipboard

Zihan Zhou, Chong Li, Xinyi Chen, Shuo Wang, Yu Chao, Zhili Li, Haoyu Wang, Qi Shi, Zhixing Tan, Xu Han, Xiaodong Shi, Zhiyuan Liu, Maosong Sun

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts (2025.findings-acl)

Challenge:	Existing studies have focused on extending the context length of large language models (LLMs) due to their quadratic computational complexity and a lack of high-quality long training examples, most LLMs are trained with a limited window size.
Approach:	They propose a training-free framework that enables large language models to effectively process long texts using a divide-and-conquer strategy for comprehensive document understanding.
Outcome:	The proposed framework outperforms open-source and commercial long-context LLMs and is compatible with several models.

Copied to clipboard

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao

Challenge:	Large Language Models (LLMs) have outstanding performance by learning a large number of model parameters on large amounts of data.
Approach:	They propose a method of grouping and pruning similar experts to improve the model’s parameter efficiency by a range of natural language tasks.
Outcome:	The proposed method outperforms other model pruning methods on a range of natural language tasks.

LiST: Lite Prompted Self-training Makes Parameter-efficient Few-shot Learners (2022.findings-naacl)

Copied to clipboard

Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, Jianfeng Gao

Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (N19-1)

Challenge:	LiST is an efficient method for fine-tuning large pre-trained language models in few-shot learning settings.
Approach:	They propose a method for efficient fine-tuning of large pre-trained language models in few-shot settings using self-training and meta-learning.
Outcome:	The proposed method outperforms GPT-3 in-context learning by 33% on few-shot tasks.

Copied to clipboard

Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)

Challenge:	Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing tasks.
Approach:	They propose a cyclical annealing schedule which repeats the process of increasing multiple times to learn more meaningful latent codes progressively by leveraging previous learning cycles as warm re-restart.
Outcome:	The proposed method improves on a broad range of NLP tasks, including language modeling, dialog response generation and semi-supervised text classification.

Copied to clipboard

Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao

A Tale of Two Linkings: Dynamically Gating between Schema Linking and Structural Linking for Text-to-SQL Parsing (2020.coling-main)

Challenge:	Existing methods for fine-tuning pre-trained models fail to generalize to unseen data.
Approach:	They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks .
Outcome:	The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI.

Copied to clipboard

Sanxing Chen, Aidan San, Xiaodong Liu, Yangfeng Ji

Programming Languages in NLP

Challenge:	Existing methods for text-to-SQL semantic parsing require strict structured prediction due to its application scenario where the output SQL will be sent to an executor program directly.
Approach:	They propose to use schema linking and structural linking to link NL to the database schema.
Outcome:	The proposed method shows significant gains on the Spider dataset.

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization (2023.findings-emnlp)

Copied to clipboard

Lei Lin, Shuangtao Li, Yafang Zheng, Biao Fu, Shan Liu, Yidong Chen, Xiaodong Shi