Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work has examined the training dynamics of language models. |
| Approach: | They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge. |
| Outcome: | The proposed model size and training checkpoints influence convergence of language models under different seeds. |
Similar Papers
Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Fig. 1 and 2 shows that log-likelihood vectors provide a consistent representation for language models . weight permutation symmetries and architectural dependencies hinder direct comparisons between models with different learning methods or designs. |
| Approach: | They propose a log-likelihood vector for comparing language models as probability distributions . they establish a consistent scale for KL divergence across various settings . |
| Outcome: | The proposed model comparisons show that the log-likelihood space is smaller than the weight space . the proposed model compares language models across checkpoints, model sizes, quantization, fine-tuning, and layers . |
Do language models accommodate their users? A study of linguistic convergence (2026.eacl-long)
Copied to clipboard
| Challenge: | In this paper, we examine how large language models adapt their language use to the linguistic patterns of their user. |
| Approach: | They examine whether large language models exhibit linguistic convergence, a pragmatic element of human language communication, and compare their results to original human responses. |
| Outcome: | The proposed model language use is significantly different from that of humans. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Tending Towards Stability: Convergence Challenges in Small Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasing the number of parameters in language models is a common strategy to enhance performance, but smaller models often underperform compared to their larger counterparts due to their reduced representational capacity. |
| Approach: | They use the Pythia model suite to analyse the training dynamics that underlie this phenomenon. |
| Outcome: | The proposed model suite enables us to examine the training dynamics of small models. |
Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect text in training corpus are limited due to their low token probabilities. |
| Approach: | They propose a method to calibrate token probabilities for pretraining data detection by using a divergence-based calibration method. |
| Outcome: | The proposed method significantly outperforms existing methods on Chinese text on English-language benchmarks and patents. |
Learning Is Not A Race: Improving Retrieval in Language Models via Equal Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Overparametrized models trained on cross-entropy loss can overfit on noise . Fitting some tokens early reduces gradient signals in later iterations . |
| Approach: | They propose to overfit models trained on cross-entropy loss on noise . fitting some tokens early reduces gradient signals in later iterations . |
| Outcome: | The proposed approaches can be applied to large language models with longer contexts or larger embedding sizes. |
A Distributional Perspective on Word Learning in Neural Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being studied as models of human language learners. |
| Approach: | They propose a distributional approach to word learning that captures distributional knowledge and gradient preferences for the word’s appropriateness. |
| Outcome: | The proposed signatures capture knowledge of where the target word can and cannot occur as well as gradient preferences about the word’s appropriateness. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |
Experimenting with Power Divergences for Language Modeling (D19-1)
Copied to clipboard
| Challenge: | Language models are an important component in many NLP tasks, where they provide prior knowledge on the language used. |
| Approach: | They propose to use power divergences to prioritize learning on frequent or rare words . they use a sample-based objective to approximate a softmax and noise-constrained estimate . |
| Outcome: | The proposed power divergences can be used to prioritize learning on the frequent or rare words and lead to general performance improvements. |
Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. |
| Approach: | They propose to find a prompt that induces LMs to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. |
| Outcome: | The proposed model is able to generate a distribution as close as possible to a target given a prompt, and it can be used to approximate distributions with low or high entropy. |