DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling (2021.emnlp-main)
Copied to clipboard
Baojun Wang, Zhao Zhang, Kun Xu, Guang-Yuan Hao, Yuyang Zhang, Lifeng Shang, Linlin Li, Xiao Chen, Xin Jiang, Qun Liu
| Challenge: | Existing approaches to integrate lexical knowledge into deep learning models are limited by large-scale dynamic lexicons. |
| Approach: | They propose a plug-in lexicon incorporation approach for BERT based sequence labeling tasks . they adopt word-agnostic tag embeddings to avoid re-training the representation . |
| Outcome: | The proposed framework achieves new SOTA even with large scale lexicons, the authors show . they adopt word-agnostic tag embeddings to avoid re-training the representation . |
Similar Papers
Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Chinese sequence labelling only fuse lexicon features via a shallow and random initialized sequence layer and do not integrate them into the bottom layers of BERT. |
| Approach: | They propose a Lexicon Enhanced BERT model which integrates external lexicon knowledge into BERT layers directly by a lexiccon Adapter layer. |
| Outcome: | The proposed model integrates external lexicon knowledge into BERT layers directly by a Lexicon Adapter layer. |
To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging (2020.emnlp-main)
Copied to clipboard
Kasturi Bhattacharjee, Miguel Ballesteros, Rishita Anubhai, Smaranda Muresan, Jie Ma, Faisal Ladhak, Yaser Al-Onaizan
| Challenge: | Using large amounts of unlabeled data to improve performance has become the foundation for many natural language processing tasks. |
| Approach: | They propose a task-specific semi-supervised approach that uses unlabeled data in a more task-agnostic manner. |
| Outcome: | The proposed approach achieves similar performance to BERT on a set of sequence tagging tasks with less financial and environmental impact. |
Small and Practical BERT Models for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Existing models for morphosyntactic tagging have focused on building separate models for each language or for a small group of related languages. |
| Approach: | They propose a scheme to train a single multilingual sequence labeling model that is small and fast enough to run on a CPU. |
| Outcome: | The proposed model outperforms state-of-the-art models on low-resource languages and low-level models on codemixed inputs. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
Fine-grained Knowledge Fusion for Sequence Labeling Domain Adaptation (D19-1)
Copied to clipboard
| Challenge: | Existing domain adaptation methods focus on the adaptation from the source domain to the entire target domain without considering the diversity of individual sample samples. |
| Approach: | They propose a fine-grained knowledge fusion model with the domain relevance modeling scheme to control the balance between learning from the target domain data and learning from a source domain model. |
| Outcome: | The proposed model outperforms baselines and state-of-the-art models on three sequence labeling tasks. |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
Bringing Emerging Architectures to Sequence Labeling in NLP (2026.eacl-long)
Copied to clipboard
| Challenge: | Pretrained Transformer encoders are the dominant approach to sequence labeling . however, few have been applied to sequence labels on flat or simplified tasks . |
| Approach: | They propose to use pretrained Transformer encoders to model relations across words . they find that the architectures adapt well across tagging tasks that vary in complexity . |
| Outcome: | The proposed architectures perform well across tagging tasks across languages and datasets. |
Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation (P19-1)
Copied to clipboard
| Challenge: | Pretrained contextual and non-contextual subword embeddings are available in over 250 languages, allowing massively multilingual NLP. |
| Approach: | They compare pretrained contextual and non-contextual subword embeddings with a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. |
| Outcome: | The proposed method outperforms non-contextual embeddings on multilingual named entity recognition and part-of-speech tagging. |
Jeff Da at COIN - Shared Task: BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge (D19-60)
Copied to clipboard
| Challenge: | Recent studies show that large-scale pre-training models can be effective for large datasets. |
| Approach: | They propose a method of integrating contextual embeddings with commonsense graph embeddINGs by preprocessing knowledge bases and aligning tokens between misaligned tokenization methods. |
| Outcome: | The proposed method achieves higher accuracy than BERT and scores highest without pretraining. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |