Papers by Yankai Lin

Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models (2022.coling-1)

Challenge:	Pre-trained language models (PLMs) have made impressive results in a wide range of NLP tasks.
Approach:	They propose a pre-training model with editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’.
Outcome:	The proposed model decouples the knowledge storage from model parameters with an editable and scalable key-value memory and leverages knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’.

Copied to clipboard

Zichun Yu, Tianyu Gao, Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Maosong Sun, Jie Zhou

Denoising Distantly Supervised Open-Domain Question Answering (P18-1)

Challenge:	Prompting has shown to be sample efficient compared to fine-tuning with pre-trained models.
Approach:	They propose a fully automatic prompting method that uses natural language prompts on sequence-to-sequence models and a beam search method to generate a large amount of label sequence candidates.
Outcome:	The proposed method significantly outperforms other no-manual-design methods on single label words and generates large amount of label sequence candidates.

Copied to clipboard

Yankai Lin, Haozhe Ji, Zhiyuan Liu, Maosong Sun

Question Answering

Challenge:	Existing DS-QA models ignore rich information contained in other paragraphs and are noisy . Existing systems rely on pre-identified relevant texts, which do not always exist in real-world QA scenarios.
Approach:	They propose a model which uses a paragraph selector to filter out noisy paragraphs and a reader to extract the correct answer from denoised paragraphs.
Outcome:	The proposed model can capture useful information from noisy data and achieve significant improvements on open domain question answering.

RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation (2024.emnlp-demo)

Copied to clipboard

Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, Maosong Sun

Coreferential Reasoning Learning for Language Representation (2020.emnlp-main)

Challenge:	Xia et al., 2018) demonstrate that a large language model can generate and maintain high-quality code documentation.
Approach:	They propose a large language model powered open-source framework for generating, maintaining, and updating code documentation.
Outcome:	The proposed framework generates high-quality documentation for the entire project.

Copied to clipboard

Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, Zhiyuan Liu

Exploring Mode Connectivity for Pre-trained Language Models (2022.emnlp-main)

Challenge:	Existing language representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse.
Approach:	They propose a language representation model that captures coreferential relations in context.
Outcome:	The proposed model can achieve significant improvements on downstream NLP tasks while maintaining comparable performance to baseline models on other common NLP task.

Copied to clipboard

Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, Jie Zhou

Packed Levitated Marker for Entity and Relation Extraction (2022.acl-long)

Challenge:	Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found.
Approach:	They investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path.
Outcome:	The proposed model can be used to find low-loss paths between two minima, and to understand how their mode connectivity affects their task knowledge.

Copied to clipboard

Deming Ye, Yankai Lin, Peng Li, Maosong Sun

Knowledge Inheritance for Pre-trained Language Models (2022.naacl-main)

Challenge:	Existing work on entity and relation extraction ignores the interrelation between spans . a novel approach to extract better span representations from pre-trained languages is needed .
Approach:	They propose a span representation approach that packs Levitated Markers to consider interrelation between spans.
Outcome:	The proposed model improves on baselines on six NER benchmarks and achieves a 4.1%-4.3% strict relation F1 improvement with higher speed over previous state-of-the-art models.

Copied to clipboard

Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

MAVEN: A Massive General Domain Event Detection Dataset (2020.emnlp-main)

Challenge:	Existing large-scale pre-trained language models are mainly trained from scratch individually, ignoring that many well-taught PLMs are available.
Approach:	They propose a pre-training framework called knowledge inheritance and propose auxiliary supervision to efficiently learn larger PLMs.
Outcome:	The proposed framework can be used to train large-scale language models with huge parameters and a large dataset can be adapted to domain adaptation and knowledge transfer.

Copied to clipboard

Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, Jie Zhou

MAVEN-ARG: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation (2024.acl-long)

Challenge:	Existing datasets exhibit data scarcity and limited coverage of general-domain events.
Approach:	They present a MAssive eVENt detection dataset which contains 4,480 Wikipedia documents and 168 event types.
Outcome:	The proposed dataset shows that existing methods cannot achieve promising results on the small datasets.

Copied to clipboard

Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, Juanzi Li

ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions (2024.findings-acl)

Challenge:	Existing datasets for event understanding have limited coverage due to complexity of tasks.
Approach:	They propose a dataset that augments MAVEN datasets with event argument annotations . they propose 98,591 events and 290,613 arguments obtained with laborious human annotation .
Outcome:	The proposed dataset is the first all-in-one dataset supporting event detection, event argument extraction, and event relation extraction.

Copied to clipboard

Honglin Lin, Siyu Li, Guoshun Nan, Chaoyue Tang, Xueting Wang, Jingxin Xu, Rong Yankai, Zhouzhili Zhouzhili, Yutong Gao, Qimei Cui, Xiaofeng Tao

Visual Data in NLP

Challenge:	Existing approaches to image retrieval from contextual descriptions (IRCD) lag behind human performance in IRCD.
Approach:	They propose a method that relies on a doubly contextual alignment scheme for challenging IRCD.
Outcome:	The proposed method can yield comparable results with GPT-4V, despite fewer parameters.

More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction (2020.aacl-main)

Copied to clipboard

Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, Maosong Sun

Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents (2024.acl-long)

Challenge:	Existing methods for extracting relational facts from text have been successful . but with explosion of Web text, human knowledge is increasing drastically .
Approach:	They propose to improve relation extraction methods to extract relational facts from text . they analyze existing methods and show promising directions towards more powerful RE .
Outcome:	The proposed methods can extract relational facts from text, but they are still lacking in the current field.

Copied to clipboard

Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning (2025.emnlp-demos)

Challenge:	Current language model-driven agents lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions.
Approach:	They propose a benchmark to inspect users’ implicit intentions through explicit queries and a model expert as the upstream in agent design to enhance user-agent interaction.
Outcome:	The proposed approach excels at identifying vague user tasks, recovering and summarizing critical missing information, setting precise and necessary agent execution goals, and minimizing redundant tool usage, thus boosting overall efficiency.

Copied to clipboard

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Xie Xie, Wei Zhou, Wang Xu, Zhou Su, Zhongwu Zhai, Xiaoming Liu, null Meiyudong, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022.acl-long)

Challenge:	Large language model agents have enabled GUI-based automation, but their deployment is limited by noisy data, poor generalization, and lack of support for non-English GUIs.
Approach:	They propose an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction.
Outcome:	The proposed GUI agent achieves promising performance on five public benchmarks and proposed Chinese benchmark CAGUI.

Copied to clipboard

Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, Minlie Huang

Low-Resource NLP

Challenge:	Existing reference-free metrics have obvious limitations for evaluating controlled text generation models.
Approach:	They propose an unsupervised reference-free metric which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks.
Outcome:	The proposed metric has higher correlations with human judgments while obtaining better generalization of evaluating generated texts from different models and with different qualities.

Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification (2026.findings-acl)

Copied to clipboard

Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment (2024.emnlp-main)

Challenge:	Experimental results show that LENS outperforms GRPO in delivering higher performance and faster convergence.
Approach:	They propose a framework that purifies prompts by identifying and removing interference tokens and then transfers successful rollouts to supervise policy optimization on original noisy prompts.
Outcome:	The proposed framework outperforms GRPO in the real-world, with a 3.88% gain and speedup.

Copied to clipboard

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

Challenge:	Existing algorithms for achieving optimal alignment are mostly unidirectional . a recent study suggests that large language models can be ground with evident preferences .
Approach:	They propose to ground large language models with evident preferences . they propose to use controllable preference optimization to specify different objectives .
Outcome:	The proposed models can provide responses that match various preferences among the ”3H” desiderata.

Large Language Model-based Human-Agent Collaboration for Complex Task Solving (2024.findings-emnlp)

Copied to clipboard

Xueyang Feng, Zhi-Yuan Chen, Yujia Qin, Yankai Lin, Xu Chen, Zhiyuan Liu, Ji-Rong Wen

Forest Before Trees: Latent Superposition for Efficient Visual Reasoning (2026.acl-long)

Challenge:	Recent advances in large language models have led to the development of LLM-based autonomous agents.
Approach:	They propose a Reinforcement Learning-based Human-Agent Collaboration method which trains a policy model to determine the most opportune stages for human intervention within the task-solving process.
Outcome:	The proposed method improves human-agent collaboration significantly through well-planned, limited human intervention.

Copied to clipboard

Yubo Wang, Juntian Zhang, Yichen Wu, Yankai Lin, Nils Lukas, Yuhan Liu

Visual Data in NLP

Challenge:	Recent latent reasoning methods suffer from a bandwidth bottleneck . explicit textual rationales suffer from premature semantic collapse .
Approach:	They propose a new paradigm that reformulates visual deduction via Dynamic Windowed Alignment Learning.
Outcome:	The proposed paradigm achieves state-of-the-art performance among latent reasoning methods surpassing the strong baseline Monet by 5.03% on average.

TrendSim: Simulating Trending Topics in Social Media Under Poisoning Attacks with LLM-based Multi-agent System (2025.findings-naacl)

Copied to clipboard

Zeyu Zhang, Jianxun Lian, Chen Ma, Yaning Qu, Ye Luo, Lei Wang, Rui Li, Xu Chen, Yankai Lin, Le Wu, Xing Xie, Ji-Rong Wen

TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference (2021.naacl-main)

Challenge:	Trending topics bring in a new channel for poisoning attacks, resulting in negative impacts on society.
Approach:	They propose an LLM-based multi-agent system to simulate trending topics in social media . they propose a time-aware interaction mechanism, centralized message dissemination, and an interactive system .
Outcome:	The proposed system simulates trending topics under poisoning attacks on social media platforms.

Copied to clipboard

Deming Ye, Yankai Lin, Yufei Huang, Maosong Sun

Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning (2023.findings-acl)

Challenge:	Existing pre-trained language models (PLMs) are expensive in inference, making them impractical in resource-limited real-world applications.
Approach:	They propose a dynamic token reduction approach to accelerate PLMs' inference by adapting the layer number of each token to avoid redundant calculation.
Outcome:	The proposed approach speeds up BERT by 2-5 times and improves performance in long-text tasks with less computation.

Copied to clipboard

Weize Chen, Xu Han, Yankai Lin, Zhiyuan Liu, Maosong Sun, Jie Zhou

Disentangle-based Continual Graph Representation Learning (2020.emnlp-main)

Challenge:	Existing methods for tuning pre-trained language models ignore the running cost and only optimize the terminal cost.
Approach:	They propose to use stochastic bridges to regularize intermediate states and use regularization as running cost of PETs.
Outcome:	The proposed methods can be used to tune large pre-trained language models . they can be compared to full-parameter fine-tuning by tuning a small number of parameters .

Copied to clipboard

Xiaoyu Kou, Yankai Lin, Shaobo Liu, Peng Li, Jie Zhou, Yan Zhang

Emergent Modularity in Pre-trained Transformers (2023.findings-acl)

Challenge:	Existing graph embedding methods overlook streaming nature of incoming data in real-world applications.
Approach:	They propose a disentangle-based continual graph representation learning framework inspired by the human’s ability to learn procedural knowledge.
Outcome:	The proposed framework outperforms state-of-the-art continual graph representation learning framework and alleviate catastrophic forgetting problem.

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou

GAM: Hierarchical Graph-based Agentic Memory for LLM Agents (2026.acl-long)

Challenge:	Existing studies on pre-trained Transformers show that they learn fine-grained neuron functions.
Approach:	They examine the presence of modularity in pre-trained Transformers . they focus on Mixture-of-Experts, a promising candidate for modularity .
Outcome:	The proposed structure stabilizes at the early stage, which is faster than neuron stabilization.

Copied to clipboard

Zhaofen Wu, Hanrong Zhang, Fulin Lin, Wujiang Xu, Xinran Xu, Yankai Chen, Henry Peng Zou, Shaowen Chen, Weizhi Zhang, Xue Liu, Philip S. Yu, Hongwei Wang

Structured Data in NLP

Challenge:	Current unified stream-based memory systems facilitate context updates but remain vulnerable to interference from transient noise.
Approach:	They propose a hierarchical Graph-based Agentic Memory framework that explicitly decouples memory encoding from consolidation to resolve conflict between rapid context perception and stable knowledge retention.
Outcome:	The proposed framework outperforms state-of-the-art benchmarks on LoCoMo and LongDialQA.

ICLEval: Evaluating In-Context Learning Ability of Large Language Models (2025.coling-main)

Copied to clipboard

Wentong Chen, Yankai Lin, ZhenHao Zhou, HongYun Huang, YanTao Jia, Zhao Cao, Ji-Rong Wen

CLEVE: Contrastive Pre-training for Event Extraction (2021.acl-long)

Challenge:	Existing evaluation frameworks focus on language abilities and knowledge, often overlooking the assessment of ICL ability.
Approach:	They propose to evaluate the ICL ability of Large Language Models (LLMs) using the ICLEval benchmark.
Outcome:	The proposed benchmark demonstrates that ICL ability is universally present in different LLMs and model size is not the sole determinant of ICL efficacy.

Copied to clipboard

Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, Jie Zhou

LLM-Based Multi-Agent Systems are Scalable Graph Generative Models (2025.findings-acl)

Challenge:	Existing EE methods do not model event characteristics from large unsupervised data.
Approach:	They propose a contrastive pre-training framework for event extraction to better learn event knowledge from large unsupervised data and their semantic structures.
Outcome:	The proposed framework improves on ACE 2005 and MAVEN datasets on event extraction tasks.

Copied to clipboard

Jiarui Ji, Runlin Lei, Jialing Bi, Zhewei Wei, Xu Chen, Yankai Lin, Xuchen Pan, Yaliang Li, Bolin Ding

Adversarial Multi-lingual Neural Relation Extraction (C18-1)

Challenge:	Social graphs are mathematical structures stem from pairwise interactions between entities through nodes and edges.
Approach:	They propose a framework for dynamic, text-attributed social graph generation that simulates the temporal node and edge generation processes for zero-shot social graphs.
Outcome:	The proposed framework improves macroscopic graph structure metrics by 11% . the proposed model can generate graphs with up to 100,000 nodes or 10 million edges .

Copied to clipboard

Xiaozhi Wang, Xu Han, Yankai Lin, Zhiyuan Liu, Maosong Sun

Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023.emnlp-main)

Challenge:	Existing models cannot capture consistency and diversity of relation patterns in different languages.
Approach:	They propose an adversarial multi-lingual neural relation extraction model which considers consistency and diversity among languages.
Outcome:	The proposed model outperforms the state-of-the-art models on real-world datasets.

Copied to clipboard

Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun

Cross-lingual Lexical Sememe Prediction (D18-1)

Challenge:	Parameter-efficient tuning (PET) methods can drive large pre-trained language models by training only minimal parameters.
Approach:	They propose a parameter-efficient tuning method that is compatible with a tunable module and uses a random number generator to optimize fewer table parameters.
Outcome:	The proposed method is compatible with a tunable module and tested on 11 NLP tasks.

Copied to clipboard

Fanchao Qi, Yankai Lin, Maosong Sun, Hao Zhu, Ruobing Xie, Zhiyuan Liu

Distilling Rule-based Knowledge into Large Language Models (2025.coling-main)

Challenge:	Sememes are defined as the minimum semantic units of human languages . but most languages do not have sememe-based linguistic knowledge bases . a new framework is proposed to predict sememes for words in other languages based on semems .
Approach:	They propose a framework to model correlations between sememes and multi-lingual words in low-dimensional semantic space for sememe prediction.
Outcome:	The proposed model improves on baseline methods on real-world datasets.

Copied to clipboard

Wenkai Yang, Yankai Lin, Jie Zhou, Ji-Rong Wen

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction (2022.emnlp-main)

Challenge:	Recent advances in large language models have broadened their applicability across diverse realworld scenarios.
Approach:	They propose to encode rule-based knowledge into large language models by using strong in-context abilities to extract the knowledge from the textual rules and then explicitly encode it into the parameters of LLMs.
Outcome:	The proposed learning paradigm is much more efficient than example-based learning in both sample size and generalization ability.

Copied to clipboard

Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts (2022.findings-acl)

Challenge:	Existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions.
Approach:	They construct a large-scale human-annotated ERE dataset with improved annotation schemes to address these drawbacks.
Outcome:	The proposed dataset is larger than existing datasets of all the ERE tasks by at least an order of magnitude.

Copied to clipboard

Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

AgentRM: Enhancing Agent Generalization with Reward Modeling (2025.acl-long)

Challenge:	Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge.
Approach:	They propose to convert a model into its MoE version with the same parameters and build expert routers to decide which experts will be used for each input.
Outcome:	The proposed model can use 10% to 30% of FFN parameters while maintaining over 95% original performance.

Copied to clipboard

Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, Maosong Sun

DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction (P19-1)

Challenge:	Existing LLM-based agents have strong performance on held-in tasks, but their generalizability to unseen tasks remains poor.
Approach:	They propose a reward-based generalizable reward model to guide the policy model for effective test-time search.
Outcome:	The proposed agentRM outperforms existing agents on held-in tasks by 8.8 points on average.

Copied to clipboard

Shun Zheng, Xu Han, Yankai Lin, Peilin Yu, Lu Chen, Ling Huang, Zhiyuan Liu, Wei Xu

On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)

Challenge:	Existing methods for labeling relational facts require significant expert labor to write relation-specific patterns, which makes them too sophisticated to generalize quickly.
Approach:	They propose a neural pattern diagnosis framework that can summarize and refine relation-specific patterns with human experts in the loop.
Outcome:	The proposed framework can summarize and refine high-quality relational patterns from noise data with human experts in the loop.

Copied to clipboard

Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou

Plug-and-Play Knowledge Injection for Pre-trained Language Models (2023.acl-long)

Challenge:	Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing.
Approach:	They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability.
Outcome:	The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing.

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Beyond the Surface: Measuring Self-Preference in LLM Judgments (2025.emnlp-main)

Challenge:	Existing knowledge injection methods are not suitable for enhancing pre-trained language models with external knowledge bases.
Approach:	They propose a plug-and-play knowledge injection method where knowledge bases are injected into frozen existing downstream models by a knowledge plugin.
Outcome:	The proposed method improves the performance of knowledge injection on knowledge-driven tasks while keeping model parameters frozen.

Copied to clipboard

Zhi-Yuan Chen, Hao Wang, Xinyu Zhang, Enrui Hu, Yankai Lin

Towards Tool Use Alignment of Large Language Models (2024.emnlp-main)

Challenge:	Existing methods measure self-preference bias by comparing the scores a judge model assigns to its own responses with those assigned to other models.
Approach:	They propose to use gold judgments as proxies for the actual quality of responses . they propose to measure self-preference bias as the difference between the judge model's own and other models' scores .
Outcome:	The proposed method can assess self-preference bias across large language models . it uses gold judgments as proxies for the ground truth scores of the judge model .

Copied to clipboard

Zhi-Yuan Chen, Shiqi Shen, Guangyao Shen, Gong Zhi, Xu Chen, Yankai Lin

LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models (2026.acl-long)

Challenge:	Existing studies on tool use with LLMs focus on enhancing tool-calling ability of LLM . e.g., LLM should not answer unsafe tool use relevant instructions or insecure tool responses to ensure reliability and harmlessness.
Approach:	They propose to use supervised fine-tuning and preference learning to align LLMs with H2A principle for tool use.
Outcome:	The proposed model demonstrates that LLMs can generate truthful and helpful responses while remaining harmless.

Copied to clipboard

Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li

RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models (2021.emnlp-main)

Challenge:	Masked diffusion language models have achieved significant progress in language modeling . however, the systematic analysis and empirical validation of their alignment on general tasks remains underexplored.
Approach:	They propose a framework that analyzes the bias and variance of preference optimization loss and gradient based on Direct Preference Optimization.
Outcome:	The proposed model outperforms its SFT-only predecessor on general benchmarks . it consistently outperformed other strong language models and ARMs on general tasks .

Copied to clipboard

Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, Xu Sun

A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models (2022.acl-short)

Challenge:	Backdoor attacks are a serious threat to the safety of reusing deep neural networks (DNNs).
Approach:	They propose an efficient online defense mechanism based on robustness-aware perturbations to distinguish poisoned and clean samples to defend against backdoor attacks on natural language processing models.
Outcome:	The proposed method achieves better defending performance and lower computational costs than existing defense methods.

Copied to clipboard

Deming Ye, Yankai Lin, Peng Li, Maosong Sun, Zhiyuan Liu

XQA: A Cross-lingual Open-domain Question Answering Dataset (P19-1)

Challenge:	Existing pre-trained language models cannot recall factual knowledge of entities exhibited in large-scale corpora, especially those rare entities.
Approach:	They propose to build a pluggable Entity Lookup Table (PELT) on demand by aggregating the entity’s output representations of multiple occurrences in the corpora.
Outcome:	The proposed model can transfer entity knowledge from out-of-domain corpora into PLMs with different architectures.

Copied to clipboard

Jiahua Liu, Yankai Lin, Zhiyuan Liu, Maosong Sun

Natural Language Interfaces

Challenge:	Open-domain question answering aims to answer questions through text retrieval and reading comprehension . but, the success of these models relies on a massive volume of training data, which is not available in other languages . a new dataset aims at investigating cross-lingual OpenQA .
Approach:	They propose to use a dataset for cross-lingual OpenQA research to test models . they use XQA dataset to train models with large volumes of labeled data .
Outcome:	The proposed model achieves best results in almost all target languages while the performance is lower than that of English.

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models (2023.findings-emnlp)

Copied to clipboard

Weize Chen, Xiaoyue Xu, Xu Han, Yankai Lin, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou

NumNet: Machine Reading Comprehension with Numerical Reasoning (D19-1)

Challenge:	Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments.
Approach:	They propose a method to enhance the inference efficiency of parameter-shared PLMs by pre-training models that can achieve even greater acceleration.
Outcome:	The proposed method improves inference efficiency on autoregressive and autoencoding models.

Copied to clipboard

Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu

Reasoning

Challenge:	Existing numerical MRC models are weak in numerical reasoning, such as addition, subtraction, sorting and counting.
Approach:	They propose a numerical MRC model that integrates numerical reasoning into existing MRC models and achieves an EM-score of 64.56% on the DROP dataset.
Outcome:	The proposed model outperforms all existing machine reading comprehension models by considering the numerical relations among numbers on the DROP dataset.

Graph Neural Networks with Generated Parameters for Relation Extraction (P19-1)

Copied to clipboard

Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, Maosong Sun

Towards Effective and Efficient Continual Pre-training of Large Language Models (2025.acl-long)

Challenge:	Existing graph neural networks can only process multi-hop relational reasoning on pre-defined graphs and cannot be directly applied in natural language relational reasoning.
Approach:	They propose a graph neural network with generated parameters using natural language sentences as inputs.
Outcome:	The proposed model can process relational reasoning on graphs and in natural language processing tasks.

Copied to clipboard

Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen

GUICourse: From General Vision Language Model to Versatile GUI Agent (2025.acl-long)

Challenge:	Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks.
Approach:	They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs.
Outcome:	The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs.

Copied to clipboard

Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun

Visual Data in NLP

Challenge:	Graphical User Interfaces (GUIs) are a pivotal medium for human-computer interaction.
Approach:	They propose a series of datasets for training visual-based GUI agents using general VLMs.
Outcome:	The proposed GUICourse datasets show that even a small-sized GUI agent performs better on GUI tasks.

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs (2025.findings-acl)

Copied to clipboard

Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu

Challenge:	Positional biases in large language models hinder their ability to process long inputs.
Approach:	They propose a benchmark to assess positional bias in large language models involving multiple pieces of relevant information.
Outcome:	The proposed benchmark assesses the performance of long-context language models by examining their models with different input lengths and tasks.

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach (2022.findings-acl)

Copied to clipboard

Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou

WebCPM: Interactive Web Search for Chinese Long-form Question Answering (2023.acl-long)

Challenge:	Pre-trained language models capture factual knowledge from massive texts . but they are still quite behind the SOTA KGC models in terms of performance .
Approach:	They propose to use open-world assumption to evaluate PLM-based knowledge graph completion models . they propose to convert each triple and its support information into natural prompt sentences .
Outcome:	The proposed model is more accurate under the open-world assumption (OWA) this setting manual checks the correctness of knowledge that is not in KGs.

Copied to clipboard

Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, Jie Zhou

Information Retrieval

Challenge:	Long-form question answering requires two procedures: information retrieval and information synthesis.
Approach:	They propose a Chinese long-form question answering dataset called WebCPM . the dataset is based on a web search interface that engages with a search engine in real time .
Outcome:	The proposed dataset generates answers that are no worse than human-written ones . the dataset is the first Chinese LFQA dataset .

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction (2021.findings-acl)

Copied to clipboard

Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation (2020.acl-main)

Challenge:	Distantly supervised relation extraction (RE) has attracted much attention in the past few years . previous methods to evaluate models manually or directly on autolabeled data have produced inaccurate evaluations .
Approach:	They propose to use distant supervision to generate large-scale autolabeled data . they build manually-annotated test sets for two DS-RE datasets and evaluate models .
Outcome:	The proposed method produces 53% wrong labels at the entity pair level in the popular NYT10 dataset.

Copied to clipboard

Qiu Ran, Yankai Lin, Peng Li, Jie Zhou

Multilinguality

Challenge:	Existing non-autoregressive neural machine translation models suffer from multi-modality problem . despite their autoregressivity, most NMT models suffer with slow decoding speed .
Approach:	They propose a semi-autoregressive model which generates a translation as a sequence of segments while each segment is predicted token-by-token.
Outcome:	The proposed model can achieve 4 times speedup while maintaining comparable performance.

Rethinking Stealthiness of Backdoor Attack against NLP Models (2021.acl-long)

Copied to clipboard

Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, Xu Sun

Fully Hyperbolic Neural Networks (2022.acl-long)

Challenge:	Existing backdoor attacks are not stealthy to system deployers or users.
Approach:	They propose a novel backdoor attack method based on negative data augmentation and modifying word embeddings that is much stealthier while maintaining pretty good attacking performance.
Outcome:	The proposed method is much stealthier while maintaining pretty good attacking performance.

Copied to clipboard

Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Challenge:	Existing hyperbolic neural networks encode features in the hyperbolical space yet formalize most of their operations in the tangent space.
Approach:	They propose a fully hyperbolic framework to build hyperbolical networks based on the Lorentz model by adapting Lorentzer transformations to formalize essential operations of neural networks.
Outcome:	The proposed framework has better performance on four NLP tasks compared with existing hyperbolic models .

Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules (2023.findings-emnlp)

Copied to clipboard

Chaojun Xiao, Yuqi Luo, Wenbin Zhang, Pengle Zhang, Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou

CURE: Critique-Driven Unified Reinforcement Learning for Test-Time Self-Improvement (2026.acl-long)

Challenge:	Large language models (LLMs) have been successful on NLP tasks but require huge parameter sizes and computational resources.
Approach:	They propose a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins.
Outcome:	The proposed method saves 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%.

Copied to clipboard

Guirong Chen, Shuqi Ye, Wenkai Yang, Shiqi Shen, Guangyao Shen, Yankai Lin

Challenge:	Existing critique-guided methods fail to equip models with the autonomous improvement capabilities required for test-time scaling.
Approach:	They propose a framework that jointly optimizes a single policy for standard solving, critiquing, and guided re-exploration.
Outcome:	The proposed framework maintains competitive single-turn performance and unlocks effective inference-time scaling.

Learning to Generate Structured Output with Schema Reinforcement Learning (2025.acl-long)

Copied to clipboard

Yaxi Lu, Haolun Li, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Zhiyuan Liu, Fangming Liu, Maosong Sun

Towards Preference Following in Tool Calling Language Agents (2026.findings-acl)

Challenge:	Recent advances in large language models have facilitated the development of intelligent applications like automatic web search (Qin et al., 2023) Several methods exist for generating JSON strings from LLMs, including Prompting but often miss certain schemas.
Approach:	They propose to use 40K different JSON schemas to assess models' ability to generate valid JSON outputs.
Outcome:	The proposed model improves both in generating JSON outputs and downstream tasks.

Copied to clipboard

Zhi-Yuan Chen, Siyu Lu, Qianlong Xie, Xingxing Wang, Yankai Lin

Exploring Backdoor Vulnerabilities of Chat Models (2025.coling-main)

Challenge:	Currently, large language model (LLM)-based agents can't follow user preferences when calling tools.
Approach:	They propose a benchmark to evaluate agents' ability to identify personalized user preferences from interaction histories and to adhere to these preferences when calling tools.
Outcome:	The proposed model achieves 51.16% accuracy on the APOLLO benchmark, while GPT-4o achieves only 51.13% accuracy.

Copied to clipboard

Wenkai Yang, Yunzhuo Hao, Yankai Lin

ROSE: Robust Selective Fine-tuning for Pre-trained Language Models (2022.emnlp-main)

Challenge:	Recent studies show that Large Language Models (LLMs) are susceptible to a security threat known as Backdoor Attack.
Approach:	They propose a backdoor attack method that distributes trigger scenarios across user inputs in different rounds and makes the backdoor be triggered only when all trigger scenarios have appeared in the historical conversations.
Outcome:	The proposed method achieves high attack success rates on chat models while maintaining normal capabilities on providing helpful responses to benign user requests.

Copied to clipboard

Lan Jiang, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Rui Jiang

From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models (2022.findings-emnlp)

Challenge:	Recent studies have highlighted the lack of adversarial robustness in pre-trained models.
Approach:	They propose a fine-tuning approach that conducts selective updates when adapting pre-trained models to downstream tasks.
Outcome:	The proposed approach improves adversarial robustness on downstream tasks . it eliminates spurious updates, leading to flatter and wider optima than the conventional method .

Copied to clipboard

Lei Li, Yankai Lin, Xuancheng Ren, Guangxiang Zhao, Peng Li, Jie Zhou, Xu Sun

ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021.acl-long)

Challenge:	Existing models for natural language processing (NLP) are fine-tuned and released for research and deployments.
Approach:	They propose a PLM reuse paradigm that merges teacher-PLM knowledge into a student model.
Outcome:	The proposed paradigm can reduce the computational cost and environmental side-effects of retraining the PLM from scratch.

Copied to clipboard

Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou

CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild (2021.emnlp-main)

Challenge:	Existing pre-training objectives do not explicitly model relational facts in text . Experimental results show that ERICA can improve typical PLMs on several language understanding tasks, including relation extraction, entity typing and question answering.
Approach:	They propose a contrastive learning framework ERICA to obtain a deep understanding of entities and relations in text.
Outcome:	The proposed framework can improve PLMs on several language understanding tasks, especially under low-resource settings.

Copied to clipboard

Yuan Yao, Jiaju Du, Yankai Lin, Peng Li, Zhiyuan Liu, Jie Zhou, Maosong Sun

Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL (2025.findings-acl)

Challenge:	Existing relation extraction methods focus on extracting relational facts between entity pairs within single sentences or documents.
Approach:	They present a problem of cross-document relation extraction (CRE) using human annotations.
Outcome:	The proposed dataset is the first human-annotated cross-document RE dataset . it shows that it is challenging to existing RE methods including strong BERT-based models.

Copied to clipboard

Wei Yao, Wenkai Yang, Ziqiao Wang, Yankai Lin, Yong Liu

Linguistics & Cognitive NLP

Challenge:	Weak-to-strong generalization is a promising approach to guide stronger systems, but its effectiveness is constrained by the inherent imperfections of weak model supervision.
Approach:	They propose a theoretically grounded approach that replaces forward KL divergence with reverse KL, which prioritizes high-confidence predictions.
Outcome:	The proposed approach replaces forward KL divergence with reverse KL, reducing the influence of unreliable weak supervision.

DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)

Copied to clipboard

Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun

Continual Relation Learning via Episodic Memory Activation and Reconsolidation (2020.acl-main)

Challenge:	Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored.
Approach:	They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python.
Outcome:	The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python.

Copied to clipboard

Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Challenge:	Existing methods to learn incessantly emerging novel relations are overfitting the few memorized examples of old relations, causing confusion among existing relations.
Approach:	They introduce episodic memory activation and reconsolidation (EMAR) to continual relation learning.
Outcome:	The proposed method outperforms state-of-the-art models in catastrophic forgetting old relations.

OpenKE: An Open Toolkit for Knowledge Embedding (D18-2)

Copied to clipboard

Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, Juanzi Li

Fact Discovery from Knowledge Base via Facet Decomposition (N19-1)

Challenge:	Existing knowledge embedding tools are available for embeddable knowledge graphs.
Approach:	They propose a unified framework and various fundamental models to embed knowledge graphs into a continuous low-dimensional space.
Outcome:	The toolkit and pre-trained embeddings are available on http://openke.thunlp.org/.

Copied to clipboard

Zihao Fu, Yankai Lin, Zhiyuan Liu, Wai Lam

Knowledge Representation

Challenge:	Recent years have witnessed the emergence and growth of many large-scale knowledge bases (KBs) however, there are some issues unsettled towards enriching the KBs.
Approach:	They propose a framework that decomposes the discovery problem into several facet components and an auto-encoder component to estimate some facets of the fact.
Outcome:	The proposed framework achieves promising results on a benchmark dataset.

Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Parameter-Efficient Tuning (2022.findings-emnlp)

Copied to clipboard

Jing Yi, Weize Chen, Yujia Qin, Yankai Lin, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun, Jie Zhou

Green & Sustainable NLP

Challenge:	Existing delta tuning algorithms freeze most of the parameters and only optimize minimal adaptive parameters.
Approach:	They propose to decompose DETs into a unified optimization subspace and conduct optimization within the subspace.
Outcome:	The proposed DETs achieve comparable performance to the original DET and can be transferred to another DET with non-trivial performance.

ELLE: Efficient Lifelong Pre-training for Emerging Data (2022.findings-acl)

Copied to clipboard

Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

SRAP-Agent: Simulating and Optimizing Scarce Resource Allocation Policy with LLM-based Agent (2024.findings-emnlp)

Challenge:	Existing pre-trained language models are typically trained with static data, ignoring that streaming data of various sources may continuously grow.
Approach:	They propose to use function preserved model expansion to expand existing PLM's width and depth to improve efficiency of knowledge acquisition.
Outcome:	The proposed model improves pre-training efficiency and performance over existing models on BERT and GPT.

Copied to clipboard

Jiarui Ji, Yang Li, Hongtao Liu, Zhicheng Du, Zhewei Wei, Qi Qi, Weiran Shen, Yankai Lin

Learning from Context or Names? An Empirical Study on Neural Relation Extraction (2020.emnlp-main)

Challenge:	Existing research on the allocation of public scarce resources has limitations due to data scarcity and data scariness.
Approach:	They propose a framework that integrates Large Language Models into economic simulations . they conduct extensive policy simulation experiments to verify the framework's effectiveness .
Outcome:	The proposed framework bridges the gap between theoretical models and real-world dynamics by integrating large language models into economic simulations.

Copied to clipboard

Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, Jie Zhou

Plug-and-Play Document Modules for Pre-trained Models (2023.acl-long)

Challenge:	Existing datasets may leak shallow heuristics via entity mentions, thus contributing to the high performance on RE benchmarks.
Approach:	They propose an entity-masked contrastive framework for relation extraction to gain a deeper understanding on textual context and type information while avoiding rote memorization of entities.
Outcome:	The proposed framework improves the effectiveness and robustness of neural models in different RE scenarios.

Copied to clipboard

Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao Cao, Maosong Sun

On Length Divergence Bias in Textual Matching Models (2022.findings-acl)

Challenge:	Large-scale pre-trained models have been widely adopted for document-oriented NLP tasks, such as question answering.
Approach:	They propose to decouple document encoding from downstream tasks by introducing a document plugin into the backbone of a PTM.
Outcome:	The proposed model can encode documents once and for all across different scenarios.

Copied to clipboard

Lan Jiang, Tianshu Lyu, Yankai Lin, Meng Chong, Xiaoyong Lyu, Dawei Yin

Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub (2025.acl-long)

Challenge:	Existing deep models have been successful in textual matching tasks, but it is unclear whether they understand language or measure semantic similarity of texts.
Approach:	They propose an adversarial evaluation scheme which invalidates the length divergence bias in TM datasets.
Outcome:	The proposed method improves the robustness and generalization ability of models at the same time.

Copied to clipboard

Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Cheng Qian, Zihe Wang, Yujia Qin, Yining Ye, Yaxi Lu, Chen Qian, Zhong Zhang, Yukun Yan, Yankai Lin, Zhiyuan Liu, Maosong Sun

Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)

Challenge:	Existing approaches lack flexibility to address diverse and ever-evolving user queries in open domains.
Approach:	They propose to evaluate LLMs on open-domain knowledge that requires tools to solve diverse and ever-evolving user queries.
Outcome:	The proposed system outperforms baselines in the open domain task-solving benchmark.

Copied to clipboard

Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade (2021.findings-emnlp)

Challenge:	Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset.
Approach:	They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods .
Outcome:	The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference .

Copied to clipboard

Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

DocRED: A Large-Scale Document-Level Relation Extraction Dataset (P19-1)

Challenge:	Experimental results show that CascadeBERT can achieve an overall 15% improvement under 4x speed-up compared with existing dynamic early exiting methods on six classification tasks.
Approach:	They propose a framework which emits predictions in internal layers without passing through the entire model.
Outcome:	The proposed framework can achieve 15% improvement under 4x speed-up compared with existing methods on six classification tasks yielding more calibrated and accurate predictions.

Copied to clipboard

Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun