On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies suggest that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. |
| Approach: | They construct cloze-like masks using task-specific lexicons to explain their results . they show that the majority of performance gains come from generic masks that are not associated with the lexical . |
| Outcome: | The proposed method outperforms a classic method for unsupervised parsing. |
Similar Papers
A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. |
| Approach: | They conduct a comprehensive study on 39 pretrained MLMs to examine their model factors and their social biases. |
| Outcome: | The proposed model factors influence social biases learned by an MLM and their downstream task performance. |
Injecting structural hints: Using language models to study inductive biases in language learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examines the cognitive inductive biases that make language learning possible. |
| Approach: | They structurally bias transformer language models by pretraining on synthetic data . they then evaluate their inductive biases by fine-tuning on three different languages . |
| Outcome: | The proposed method predisposes transformer models to three types of inductive biases . it also fine-tunes the models on three typologically-distant human languages . |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |
How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is a widely used self-supervised pretraining objective. |
| Approach: | They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context. |
| Outcome: | The proposed objectives show that they should have half the complexity needed to perform comparably to MLM. |
ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to model coarse-grained linguistic information do not integrate coarse-gram information into pre-training. |
| Approach: | They propose an explicitly n-gram masking method to enhance integration of coarse-grained linguistic information into pre-training. |
| Outcome: | The proposed method outperforms existing models on English and Chinese text corpora and fine-tunes on 19 downstream tasks. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study found that pre-training can teach language models to rely on hierarchical syntactic features . aaron ramirez: we find that pretraining on simpler language induces a hierarchic bias . |
| Approach: | They find that pre-training can teach language models to rely on hierarchical syntactic features . authors: this suggests that in cognitively plausible language acquisition settings, models may be more data-efficient . |
| Outcome: | a recent study shows that pre-training can teach language models to rely on hierarchical features . the findings suggest that in plausible language acquisition settings, language models may be more data-efficient than previously thought . |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases (2025.acl-long)
Copied to clipboard
| Challenge: | Pretraining language models on formal language can improve their acquisition of natural language, but it is computationally limited. |
| Approach: | They hypothesize that effective transfer occurs when the formal language captures hierarchical dependencies present in natural language and remains within the computational limitations of the model architecture. |
| Outcome: | The proposed model can achieve lower loss on natural language and better generalization with a 33% smaller token budget than training on a matched amount of natural language. |