NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding. |
| Approach: | They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
| Outcome: | The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
Similar Papers
Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown the importance of training contextualised word embedding models on the domain of the target task of interest. |
| Approach: | They propose a masking strategy which adversarially masks out those tokens which are harder to reconstruct by the underlying MLM. |
| Outcome: | The proposed training strategy outperforms random masking on six unsupervised domain adaptation tasks and achieves up to +1.64 F1 score improvements. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
Data Efficient Masked Language Modeling for Vision and Language (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. |
| Approach: | They propose a masking strategy that masks tokens with a 15% probability for text-only data. |
| Outcome: | The proposed masking strategy outperforms the baseline model on a prompt-based probing task designed to elicit image objects. |
DMLM: Descriptive Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition. |
| Approach: | They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition. |
| Outcome: | The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling. |
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing techniques for unsupervised domain adaptation (UDA) are limited by domain shift, which leads to performance degradation. |
| Approach: | They propose a fine-tuning procedure that uses a mixed classification and Masked Language Model loss to adapt to the target domain distribution in a robust and sample efficient manner. |
| Outcome: | The proposed procedure can adapt to the target domain distribution in a robust and sample efficient manner. |
Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevailing practice for adapting language models (LMs) to new domains. |
| Approach: | They propose a mask specific language model that weights the importance of domain-specific terms during fine-tuning to avoid insensitivity. |
| Outcome: | The proposed approach outperforms advanced masking strategies such as span- and PMI-based masking. |
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance . |
| Approach: | They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible . |
| Outcome: | The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes . |
Masked Language Model Scoring (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained masked language models require finetuning for most tasks. |
| Approach: | They evaluate pretrained masked language models out of the box via their pseudo-log-likelihood scores (PLLs) they attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 . |
| Outcome: | The proposed model outperforms autoregressive language models in a variety of tasks. |
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained MLMs can be used to generalize well to a wide range of tasks. |
| Approach: | They propose to append [MASK]s at a later layer to reduce sequence length for earlier layers. |
| Outcome: | The proposed method outperforms RoBERTa for 6 out of 8 GLUE tasks on average by 0.4%. |
Learning Better Masking for Better Language Model Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing PrLMs adopt a Random-Token Masking strategy with a fixed masking ratio and different contents are masked by an equal probability throughout the training. |
| Approach: | They propose two scheduled masking approaches that adaptively tune masking ratio and masked content in different training stages, which improves pre-training efficiency and effectiveness. |
| Outcome: | The proposed methods improve the pre-training efficiency and effectiveness on the downstream tasks. |