Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LMs) have shown promising ability on handling various downstream tasks including textual classification and question answering. |
| Approach: | They propose to use BabyBERTa to train child-directed speech without unmasking words while masking parameters to improve grammatical accuracy. |
| Outcome: | The proposed model achieves grammatical ability comparable to RoBERTa-base model, which is trained on 6,000 times more words and 15 times more parameters. |
Similar Papers
The Diminishing Returns of Masked Language Models to Science (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that masked language models can improve downstream tasks by pretraining larger models for longer on more data. |
| Approach: | They empirically evaluate the extent to which these results extend to tasks in science by using 14 domain-specific transformer-based masked language models. |
| Outcome: | The proposed model can improve on 12 scientific tasks, but not all. |
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered. |
| Approach: | They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language. |
| Outcome: | The proposed model performance is correlated with pre-training perplexity and performance. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages (2025.acl-srw)
Copied to clipboard
| Challenge: | Low-resource languages (LRLs) face significant challenges in natural language processing due to limited data. |
| Approach: | They evaluate adapter-based methods for adapting mLMs to low-resource languages . they use unstructured text and structured knowledge from ConceptNet to evaluate adapters . |
| Outcome: | The proposed methods outperform large language models and LLaMA-3 and deepSeek-R1 models on low training data. |
On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)
Copied to clipboard
Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa
| Challenge: | Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models. |
| Approach: | They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. |
| Outcome: | The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained MLMs can be used to generalize well to a wide range of tasks. |
| Approach: | They propose to append [MASK]s at a later layer to reduce sequence length for earlier layers. |
| Outcome: | The proposed method outperforms RoBERTa for 6 out of 8 GLUE tasks on average by 0.4%. |
How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is a widely used self-supervised pretraining objective. |
| Approach: | They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context. |
| Outcome: | The proposed objectives show that they should have half the complexity needed to perform comparably to MLM. |
Revisiting Pre-Trained Models for Chinese Natural Language Processing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models have shown tremendous improvements across various NLP tasks. |
| Approach: | They propose to revisit Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pretrained model series to the community. |
| Outcome: | The proposed model improves on RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |