CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020.coling-main)
Copied to clipboard
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Jun’ichi Tsujii
| Challenge: | Pre-trained language representations from Transformers have become the most popular choice for building NLP systems. |
| Approach: | They propose a new variant of BERT that drops the wordpiece tokenization system altogether . they propose 'characterBERT' module to represent entire words by consulting their characters . |
| Outcome: | The proposed model improves performance on a variety of medical domain tasks while producing robust, word-level, and open-vocabulary representations. |
Similar Papers
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |
schuBERT: Optimizing Elements of BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer based models have achieved state-of-the-art performance for many natural language processing tasks including machine translation, question-answering tasks and semantic role labeling. |
| Approach: | They propose to reduce the number of parameters of BERT to obtain a much efficient light model. |
| Outcome: | The proposed model achieves 6.6% higher average accuracy on GLUE and SQuAD datasets than the previous model with three encoder layers while having the same number of parameters. |
GhostBERT: Generate More Features with Cheap Operations for BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that some parameters in pre-trained language models can be pruned away without severe accuracy degradation. |
| Approach: | They propose a method which generates more features with very cheap operations from the remaining features and can be applied to unpruned BERT models to enhance their performance. |
| Outcome: | Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of the proposed method. |
CharBERT: Character-aware Pre-trained Language Model (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations . but these methods split a word into subword units and make it incomplete and fragile . |
| Approach: | They propose a character-aware pre-trained language model to tackle OOV problems . they construct contextual word embedding for each token from sequential character representations . |
| Outcome: | The proposed model improves on the existing models on multiple NLP benchmarks. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
Embedding Strategies for Specialized Domains: Application to Clinical Entity Recognition (P19-2)
Copied to clipboard
| Challenge: | Off-the-shelf word embeddings tend to perform poorly on texts from specialized domains such as clinical reports. |
| Approach: | They combine off-the-shelf contextual embeddings with static word2vec embedders trained on a small in-domain corpus built from task data to reach and sometimes outperform representations learned from a large corpus in the medical domain. |
| Outcome: | The proposed embedding strategies outperform representations learned from a large corpus in the medical domain. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |
What is the best recipe for character-level encoder-only modelling? (2023.acl-long)
Copied to clipboard
| Challenge: | aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. |
| Approach: | They aim to find the best way to build and train character-level BERT-like models by comparing architectural innovations with pretraining objectives. |
| Outcome: | The proposed model outperforms a token-based model on a set of evaluation tasks with a fixed training procedure. |
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a ubiquitous algorithm in the tokenization process of language models but is only based on pre-training data statistics. |
| Approach: | They propose a character-based subword module that learns the subword embedding table in pre-trained language models like BERT. |
| Outcome: | The proposed method significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark. |
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent years have witnessed the widespread use of transfer learning techniques in Natural Language Processing (NLP) |
| Approach: | They train BERT models from scratch using many configurations involving general and medical corpora. |
| Outcome: | The initial corpus only has a weak influence when these are further pre-trained on a medical corpus. |