| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
Similar Papers
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Development of Cognitive Intelligence in Pre-trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). Prior research into emergental cognitive abilities of PLMs has been path-independent to model training. |
| Approach: | They use four task categories to examine the alignment of ten popular families of PLMs and evaluate their performance to the developmental trajectories of children's thinking. |
| Outcome: | The results show that the models are more aligned to children's thinking than previous studies. |
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal . |
| Approach: | They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities. |
| Outcome: | The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. |
Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases (2025.acl-long)
Copied to clipboard
| Challenge: | Pretraining language models on formal language can improve their acquisition of natural language, but it is computationally limited. |
| Approach: | They hypothesize that effective transfer occurs when the formal language captures hierarchical dependencies present in natural language and remains within the computational limitations of the model architecture. |
| Outcome: | The proposed model can achieve lower loss on natural language and better generalization with a 33% smaller token budget than training on a matched amount of natural language. |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent success of pre-trained language models has spurred widespread interest in their capabilities. |
| Approach: | They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls . |
| Outcome: | The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models perpetuate biases originating in their training corpus to downstream models. |
| Approach: | They focus on the representations of given names in pre-trained language models and show that name perturbation can have an effect on downstream tasks. |
| Outcome: | The proposed model can be used to model the representation of given names in pre-trained language models on reading comprehension probes where name perturbation changes the model answers. |
A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. |
| Approach: | They propose to compare multilingual pretraining objectives in a controlled methodological environment with multilingual models. |
| Outcome: | The proposed model outperforms existing models in 6 languages and demonstrates that multilingual translation is an effective pretraining objective under the right conditions. |
Pretraining Language Models for Diachronic Linguistic Change Discovery (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as knowledge discovery tools . historical linguistics and literary studies often construct arguments on the basis of distinctions between phenomena like time-period or genre. |
| Approach: | They propose to use LLMs to train large language models over modest historical corpora without allowing contamination from anachronistic data. |
| Outcome: | The proposed model better respects historical divisions and is more computationally efficient compared to the standard approach of fine-tuning an existing LLM. |