SKD-NER: Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning

Challenge:	Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones.
Approach:	They propose a model that leverages knowledge distillation to retain memory and employs reinforcement learning strategies to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting.
Outcome:	The proposed model outperforms state-of-the-art models on two benchmark datasets showing that it significantly improves the performance of the CL-NER task.

Similar Papers

Overview

Challenge

Approach

Outcome

Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples (2022.findings-acl)

Copied to clipboard

Yu Xia, Quan Wang, Yajuan Lyu, Yong Zhu, Wenhao Wu, Sujian Li, Dai Dai

Language Models

Challenge:	Existing methods for named entity recognition classify mentions into fixed set of predefined entity types but in many real-world scenarios, new entity types are incrementally involved.
Approach:	They propose a two-stage framework Learn-and-Review for continual named entity recognition to alleviate inter-type confusion.
Outcome:	The proposed framework outperforms the state-of-the-art method on CoNLL-03 and OntoNotes-5.0.

Continual Named Entity Recognition without Catastrophic Forgetting (2023.emnlp-main)

Copied to clipboard

Duzhen Zhang, Wei Cong, Jiahua Dong, Yahan Yu, Xiuyi Chen, Yonggang Zhang, Zhen Fang

Information Extraction & Text Mining

Challenge:	Named Entity Recognition (CNER) is a burgeoning area of research . a new paradigm has ushered NER into a non-entity type at the current step t .
Approach:	They propose a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones.
Outcome:	The proposed method outperforms state-of-the-art approaches on ten CNER settings using three datasets.

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition (2022.emnlp-main)

Copied to clipboard

Junhao Zheng, Zhanxian Liang, Haibin Chen, Qianli Ma

Language Models

Challenge:	Existing methods for Named Entity Recognition (NER) are not able to learn Other-Class in the same way as new entity types.
Approach:	They propose a unified causal framework to retrieve causality from new entity types and Other-Class.
Outcome:	The proposed method outperforms the state-of-the-art method on three benchmark datasets.

Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition (2024.findings-acl)

Copied to clipboard

Yahan Yu, Duzhen Zhang, Xiuyi Chen, Chenhui Chu

Language Models

Challenge:	Existing methods for Named Entity Recognition (CNER) use knowledge distillation to retain old knowledge, but they are too expensive and fail to integrate with existing state-of-the-art models.
Approach:	They propose a weight tuning and weightfusion strategy to learn new entity types while mitigating catastrophic forgetting of old models.
Outcome:	The proposed strategies improve the performance of existing models and are model-agnostic.

Multi-Grained Knowledge Distillation for Named Entity Recognition (2021.naacl-main)

Copied to clipboard

Xuan Zhou, Xiao Zhang, Chenyang Tao, Junya Chen, Bing Xu, Wei Wang, Jing Xiao

Green & Sustainable NLP

Challenge:	Pre-trained big models have delivered top performance in Seq2seq modeling, but their deployments in real-world applications are often hindered by excessive computations and memory demands.
Approach:	They propose a distillation scheme to efficiently transfer knowledge from big models to their cheaper counterparts.
Outcome:	The proposed scheme maximizes the assimilation of knowledge from the teacher model to the student model.

Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity Recognition (2023.findings-acl)

Copied to clipboard

Zhiyuan Ma, Jintao Du, Shuheng Zhou

Semantic Text Processing

Challenge:	Named entity recognition (NER) is a task in natural language processing that aims at locating entity mentions in a given sentence and assigning them to certain types.
Approach:	They propose to use a dynamic loss function to better adapt to the changing noise during the training process and incorporate token level contrastive learning to fully utilize the noisy data.
Outcome:	The proposed method outperforms existing NER models on three benchmark datasets and outperformed existing models by significant margins.

Learning to Progressively Recognize New Named Entities with Sequence to Sequence Models (C18-1)

Copied to clipboard

Lingzhen Chen, Alessandro Moschitti

Language Models

Challenge:	Existing models for Named Entity Recognition (NER) are trained on data with the same NE label set, but they are not able to recognize previously unseen NE categories.
Approach:	They propose to use a sequence to sequence model for Named Entity Recognition (NER) and propose to reshape and re-parametrize the output layer of the first learned model to enable the recognition of new NEs.
Outcome:	The proposed model can recognize previously unseen NE categories while keeping the knowledge of previously seen categories.

Few-Shot Class-Incremental Learning for Named Entity Recognition (2022.acl-long)

Copied to clipboard

Rui Wang, Tong Yu, Handong Zhao, Sungchul Kim, Subrata Mitra, Ruiyi Zhang, Ricardo Henao

Low-Resource NLP

Challenge:	Existing models of Named Entity Recognition (NER) are trained on large datasets with predefined entity classes, but data of new classes arrives constantly. Existing work on NER relies on the assumption that there exists abundance of labeled data for the training of new class.
Approach:	They propose a few-shot class-incremental learning problem where NER model is trained with only few labeled samples of the new classes without forgetting knowledge of the old ones.
Outcome:	The proposed model improves over existing baselines by reconstructing training data of old classes and real data from the training set.

SpanNER: Named Entity Re-/Recognition as Span Prediction (2021.acl-long)

Copied to clipboard

Jinlan Fu, Xuanjing Huang, Pengfei Liu

Responsible & Trustworthy NLP

Challenge:	Recent years have seen the paradigm shift of Named Entity Recognition (NER) systems from sequence labeling to span prediction.
Approach:	They experimentally implement 154 named entity recognition models on 11 datasets and show that span prediction can serve as a system combiner to re-recognize named entities from different systems’ outputs.
Outcome:	The proposed model can be used to re-recognize named entities from different systems’ outputs.

ScdNER: Span-Based Consistency-Aware Document-Level Named Entity Recognition (2023.emnlp-main)

Copied to clipboard

Ying Wei, Qi Li

Information Extraction & Text Mining

Challenge:	Named entity recognition (NER) is an important task for many natural language processing applications.
Approach:	They propose to fuse global features of tokens via word-based key-value memory to produce documentlevel encoding for token label prediction.
Outcome:	The proposed model can produce consistent and consistent predictions on word level with reduced impact of non-entity sequences and adaptive global feature fusion.