Papers by Ruobing Xie

Pruning Pre-trained Language Models Without Fine-Tuning (2023.acl-long)

Challenge:	Existing datasets for event understanding have limited coverage due to complexity of tasks.
Approach:	They propose a dataset that augments MAVEN datasets with event argument annotations . they propose 98,591 events and 290,613 arguments obtained with laborious human annotation .
Outcome:	The proposed dataset is the first all-in-one dataset supporting event detection, event argument extraction, and event relation extraction.

Copied to clipboard

Ting Jiang, Deqing Wang, Fuzhen Zhuang, Ruobing Xie, Feng Xia

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models (2026.findings-acl)

Challenge:	Existing methods to prune Pre-trained Language Models (PLMs) are overparameterized and require fine-tuning.
Approach:	They propose a pruning method that uses first-order pruning to prune PLMs while fine-tuning the remaining weights.
Outcome:	The proposed method outperforms first-order pruning and zero-order methods at sparsity levels.

Copied to clipboard

Hengyuan Zhang, Zhihao Zhang, Ercong Nie, Mingyang Wang, Zunhai Su, Yiwei Wang, Qianli Wang, Shuzhou Yuan, Xufeng Duan, Qibo Xue, Zeping Yu, Chenming Shang, Xiao Liang, Jing Xiong, Hui Shen, Chaofan Tao, Zhengwu Liu, Senjie Jin, Zhiheng Xi, Dongdong Zhang, Sophia Ananiadou, Tao Gui, Ruobing Xie, Hayden Kwok-Hay So, Hinrich Schuetze, Xuanjing Huang, Qi Zhang, Ngai Wong

Explainability & Interpretability in NLP

Challenge:	Existing literature on mechanistic interpretation (MI) treats it as an observational science, leaving practical applications underexplored.
Approach:	They propose a survey structured around the pipeline to identify and improve MI models.
Outcome:	The proposed framework enables tangible improvements in Alignment, Capability, and Efficiency.

Continuous Speech Tokenizer in Text To Speech (2025.findings-naacl)

Copied to clipboard

Yixing Li, Ruobing Xie, Xingwu Sun, Yu Cheng, Zhanhui Kang

Exploring Forgetting in Large Language Model Pre-Training (2025.acl-long)

Challenge:	Autoregressive modeling is a common method for processing language sequences and is effective in token prediction.
Approach:	They propose a text-to-speech model based on continuous speech tokens and a continuous tokenizer for speech compression.
Outcome:	The proposed model has better continuity and higher estimated Mean Opinion Scores (MoS) this is attributed to better information preservation rate across low and high frequencies in the frequency domain.

Copied to clipboard

Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, Zhanhui Kang

Meta-Information Guided Meta-Learning for Few-Shot Relation Classification (2020.coling-main)

Challenge:	Existing research on task-level forgetting in LLMs has focused on pretraining . but, there is limited attention to finer-grained forgetting during training .
Approach:	They investigated the existence and measurement of forgetting in pre-training . they examined low-cost, straightforward methods to mitigate forgetting during the pre- training phase .
Outcome:	The proposed methods could be used to mitigate forgetting during the pre-training phase and offer insights into the dynamics of forgetting.

Copied to clipboard

Bowen Dong, Yuan Yao, Ruobing Xie, Tianyu Gao, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

Incorporating Global Information in Local Attention for Knowledge Representation Learning (2021.findings-acl)

Challenge:	Existing meta-learning models rely on implicit instance statistics and are unreliability and weak interpretability.
Approach:	They propose a meta-information guided meta-learning framework that uses semantics to guide meta- learning . experimental results demonstrate the effectiveness of the proposed framework .
Outcome:	The proposed framework can establish connections between instance-based information and semantic-based data, enabling faster initialization and adaptation.

Copied to clipboard

Yu Zhao, Han Zhou, Ruobing Xie, Fuzhen Zhuang, Qing Li, Ji Liu

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment (2024.emnlp-main)

Challenge:	Graph Attention Networks (GATs) are a promising model that takes advantage of localized attention mechanism to perform knowledge representation learning (KRL) on graph-structure data.
Approach:	They propose to incorporate global information into the GAT family of models by using an attention-based global random walk algorithm.
Outcome:	Experimental results on KG entity prediction against the state-of-the-arts demonstrate the effectiveness of the proposed model.

Copied to clipboard

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

Challenge:	Existing algorithms for achieving optimal alignment are mostly unidirectional . a recent study suggests that large language models can be ground with evident preferences .
Approach:	They propose to ground large language models with evident preferences . they propose to use controllable preference optimization to specify different objectives .
Outcome:	The proposed models can provide responses that match various preferences among the ”3H” desiderata.

Multimodal Clickbait Detection by De-confounding Biases Using Causal Representation Inference (2024.emnlp-main)

Copied to clipboard

Jianxing Yu, Shiqi Wang, Han Yin, Zhenlong Sun, Ruobing Xie, Bo Zhang, Yanghui Rao

Representation Learning

Challenge:	a new method to detect clickbait posts on the Web is needed to detect such posts.
Approach:	They propose a method to detect clickbait posts on the Web using latent factors . they use features in multiple modalities to characterize the posts and causal inference to eliminate noise .
Outcome:	The proposed method can detect clickbait posts on popular social media platforms with good generalization ability.

Prompt Tuning for Discriminative Pre-trained Language Models (2022.findings-acl)

Copied to clipboard

Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, Jianyong Wang

AuriSRec: Adversarial User Intention Learning in Sequential Recommendation (2024.findings-emnlp)

Challenge:	Recent studies have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing tasks.
Approach:	They propose a prompt tuning framework that reformulates NLP tasks into a discriminative language modeling problem.
Outcome:	The proposed framework improves on text classification and question answering tasks and prevents unstable tuning problems in low-resource settings.

Copied to clipboard

Junjie Zhang, Ruobing Xie, Wenqi Sun, Leyu Lin, Xin Zhao, Ji-Rong Wen

Responsible & Trustworthy NLP

Challenge:	Existing work focuses on capturing user implicit preferences from historical interactions and matching them with the next behavior, instead of predicting user explicit intentions.
Approach:	They propose an adversarial user intention learning approach for sequential recommendaiton . the approach explicitly predicts user current intentions by taking historical reviews as inputs .
Outcome:	The proposed approach explicitly predicts user intentions by inferring their decision-making process as explained in target reviews.

Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication (2024.findings-emnlp)

Copied to clipboard

Weize Chen, Chenfei Yuan, Jiarui Yuan, Yusheng Su, Chen Qian, Cheng Yang, Ruobing Xie, Zhiyuan Liu, Maosong Sun

Responsible & Trustworthy NLP

Challenge:	Natural language (NL) has long been the predominant format for human cognition and communication, but its utility in LLMs has not been thoroughly examined.
Approach:	They propose to allow LLMs to choose the most suitable format before reasoning or communicating, and to automate the selection process.
Outcome:	The proposed format improves reasoning efficiency and reduces token usage while maintaining communicative effectiveness.

Language Models “Grok” to Copy (2025.naacl-short)

Copied to clipboard

Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Rui Yan

LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders (2024.lrec-main)

Challenge:	We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context.
Approach:	They propose that Transformer-based language models develop copying abilities similarly to grokking . they argue that the connection between groking and context copying can improve in-context performance.
Outcome:	The proposed model development is similar to grokking, but the speed is independent of tokens trained.

Copied to clipboard

Xingwu Sun, Zhen Yang, Ruobing Xie, Fengzong Lian, Zhanhui Kang, Chengzhong Xu

Fusing Highly Specialized Language Models for Comprehensive Expertise (2025.acl-long)

Challenge:	Existing vision-language pre-training models use multi-modal encoders to encode image and text, causing noisy training corpora.
Approach:	They propose a vision-language pre-training framework with two autoencoders for efficient training . they propose masked tokens and a gated interaction mechanism to cope with noise .
Outcome:	The proposed model achieves 2.2% R@1 gains on COCO Text Retrieval and 1.1% on refCOCO+ on six datasets.

Copied to clipboard

Ning Ding, Yulin Chen, Ganqu Cui, Xingtai Lv, Weilin Zhao, Kaiyan Zhang, Ruobing Xie, Bowen Zhou, Zhiyuan Liu, Maosong Sun

Emergent Modularity in Pre-trained Transformers (2023.findings-acl)

Challenge:	Existing models that focus on language, programming code, and mathematical symbols are not able to achieve mastery of all three domains simultaneously.
Approach:	They propose to fuse highly-specialized models that are already sufficiently trained on different domains to achieve a highly-specific model.
Outcome:	The proposed model could achieve mastery of the three crucial domains simultaneously.

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou

Incorporating Chinese Characters of Words for Lexical Sememe Prediction (P18-1)

Challenge:	Existing studies on pre-trained Transformers show that they learn fine-grained neuron functions.
Approach:	They examine the presence of modularity in pre-trained Transformers . they focus on Mixture-of-Experts, a promising candidate for modularity .
Outcome:	The proposed structure stabilizes at the early stage, which is faster than neuron stabilization.

Copied to clipboard

Huiming Jin, Hao Zhu, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Fen Lin, Leyu Lin

Connecting Embeddings for Knowledge Graph Entity Typing (2020.acl-main)

Challenge:	Existing methods of lexical sememe prediction rely on external context information of words to represent meaning.
Approach:	They propose a character-enhanced sememe prediction framework for Chinese language that takes advantage of internal character information and external context information.
Outcome:	The proposed framework outperforms state-of-the-art methods on a Chinese sememe knowledge base and maintains robust performance even for low-frequency words.

Copied to clipboard

Yu Zhao, Anxiang Zhang, Ruobing Xie, Kang Liu, Xiaojie Wang

Cross-lingual Lexical Sememe Prediction (D18-1)

Challenge:	Existing knowledge graphs suffer from incompleteness and miss important facts, jeopardizing their usefulness in downstream tasks such as question answering.
Approach:	They propose a method which is trained by utilizing local typing knowledge from existing entity type assertions and global triple knowledge in KGs.
Outcome:	The proposed model favors inferences that agree with both entity type instances and triple knowledge in KGs.

Copied to clipboard

Fanchao Qi, Yankai Lin, Maosong Sun, Hao Zhu, Ruobing Xie, Zhiyuan Liu

Open Hierarchical Relation Extraction (2021.naacl-main)

Challenge:	Sememes are defined as the minimum semantic units of human languages . but most languages do not have sememe-based linguistic knowledge bases . a new framework is proposed to predict sememes for words in other languages based on semems .
Approach:	They propose a framework to model correlations between sememes and multi-lingual words in low-dimensional semantic space for sememe prediction.
Outcome:	The proposed model improves on baseline methods on real-world datasets.

Copied to clipboard

Kai Zhang, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised Data (D19-1)

Challenge:	Existing OpenRE methods cast different relation types in isolation without considering their hierarchical dependency.
Approach:	They propose a framework to establish bidirectional connections between OpenRE and relation hierarchies by integrating hierarchy information into relation representations.
Outcome:	The proposed framework outperforms state-of-the-art models on relation clustering and hierarchy expansion.

Copied to clipboard

Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

HMoE: Heterogeneous Mixture of Experts for Language Modeling (2025.emnlp-main)

Challenge:	Existing methods to extract relational facts from open domain corpora are time-consuming and human-intensive.
Approach:	They propose a framework to learn similarity metrics of relations from labeled data . they propose to transfer relational knowledge to identify novel relations in unlabeled data.
Outcome:	Experiments on two real-world datasets show that the proposed framework improves compared with state-of-the-art methods.

Copied to clipboard

An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, Weidong Han, Zhanhui Kang, Di Wang, Naoaki Okazaki, Cheng-zhong Xu

Sparsifying Mamba (2025.findings-emnlp)

Challenge:	Mixture of Experts (MoE) models use homogeneous experts with diverse capacities, resulting in a lack of expert specialization and parameter utilization.
Approach:	They propose a framework where experts differ in size and possess diverse capacities . they propose HMoE to encourage frequent activation of smaller experts .
Outcome:	The proposed framework outperforms homogeneous homogenous MoE models on evaluation benchmarks and achieves lower loss rate with fewer activated parameters.

Copied to clipboard

An Wang, Ruobing Xie, Shuaipeng Li, Xingwu Sun, Zhanhui Kang

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models (2023.findings-emnlp)

Challenge:	Existing attempts to integrate sparsification with Mamba fail to leverage Mamba's internal structure for fine-grained sparsifying.
Approach:	They propose to use Mamba to integrate sparsification into Mamba and propose a flexible and effective mechanism for parameter scalability.
Outcome:	The proposed framework can independently achieve parameter scalability and has stronger performance.

Copied to clipboard

Weize Chen, Xiaoyue Xu, Xu Han, Yankai Lin, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou

WebCPM: Interactive Web Search for Chinese Long-form Question Answering (2023.acl-long)

Challenge:	Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments.
Approach:	They propose a method to enhance the inference efficiency of parameter-shared PLMs by pre-training models that can achieve even greater acceleration.
Outcome:	The proposed method improves inference efficiency on autoregressive and autoencoding models.

Copied to clipboard

Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, Jie Zhou

Information Retrieval

Challenge:	Long-form question answering requires two procedures: information retrieval and information synthesis.
Approach:	They propose a Chinese long-form question answering dataset called WebCPM . the dataset is based on a web search interface that engages with a search engine in real time .
Outcome:	The proposed dataset generates answers that are no worse than human-written ones . the dataset is the first Chinese LFQA dataset .

The Security Threat of Compressed Projectors in Large Vision-Language Models (2025.findings-emnlp)

Copied to clipboard

Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Zhanhui Kang, Di Wang, Yu Wang

Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules (2023.findings-emnlp)

Challenge:	Mainstream VLPs have significant security implications, but their security implications have not been thoroughly examined.
Approach:	a study evaluates the security of visual language projectors by comparing them to uncompressed projector.
Outcome:	The evaluation reveals significant differences in security profiles between compressed and uncompressed projectors.

Copied to clipboard

Chaojun Xiao, Yuqi Luo, Wenbin Zhang, Pengle Zhang, Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou

Denoising Relation Extraction from Document-level Distant Supervision (2020.emnlp-main)

Challenge:	Large language models (LLMs) have been successful on NLP tasks but require huge parameter sizes and computational resources.
Approach:	They propose a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins.
Outcome:	The proposed method saves 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%.

Copied to clipboard

Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, Leyu Lin

Language Modeling with Sparse Product of Sememe Experts (D18-1)

Challenge:	Existing methods to generate auto-labeled sentences for relation extraction (RE) are difficult to extend to document-level relation extraction as noise from DS may be even multiplied in documents.
Approach:	They propose a pre-trained model which de-emphasizes noisy DS data via multiple pre-training tasks.
Outcome:	The proposed model can capture useful information from noisy data and achieve promising results on the large-scale DocRE benchmark.

Copied to clipboard

Yihong Gu, Jun Yan, Hao Zhu, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Fen Lin, Leyu Lin

QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models (2025.naacl-long)

Challenge:	Existing language modeling methods rely on large-scale text data to learn the sequential patterns of words.
Approach:	They propose to use sememes to represent the implicit semantics behind words for language modeling . they propose to employ sememe-driven language models to fine-grained semem-level semantics .
Outcome:	Experiments on language modeling and the downstream application of headline generation show the effectiveness of SDLM.

Copied to clipboard

Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui Kang, Yu Wang

Robustness in NLP

Challenge:	Currently, large vision-language models are limited in their ability to provide correct answers for multimodal tasks . however, they can still provide correct responses for multiple images associated with a single image . a query-agnostic visual attack (QAVA) provides robust adversarial examples that generate incorrect responses to unspecified and unknown questions.
Approach:	They propose a query-agnostic visual attack to create adversarial examples that generate incorrect answers to unspecified and unknown questions.
Outcome:	The proposed model improves performance on images when the question is unknown compared to known target questions .

Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation (2026.acl-long)

Copied to clipboard

Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang, Yuxuan Jiang, Chaofan Tao, Jing Xiong, Hayden Kwok-Hay So, Ruobing Xie, Angel X. Chang, Ngai Wong

Responsible & Trustworthy NLP

Challenge:	Existing studies show that stronger models are not always optimal teachers, suggesting a mismatch between the teacher’s output and the student’s learning ability.
Approach:	They propose a method that routes each prompt to its optimal teacher via a query-level router that jointly considers the student models’ learnability and teacher models’ response quality.
Outcome:	The proposed method outperforms baselines on six benchmarks including instruct tuning and math reasoning settings.

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization (2025.findings-acl)

Copied to clipboard

Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Xirong Li

Reasoning

Challenge:	Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications.
Approach:	They propose a method that uses three types of preference pairs to target hallucinations from their diverse forms and causes.
Outcome:	The proposed method surpasses most state-of-the-art methods and shows potential for further improvements.

Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)

Copied to clipboard

Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, Leyu Lin