Tending Towards Stability: Convergence Challenges in Small Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasing the number of parameters in language models is a common strategy to enhance performance, but smaller models often underperform compared to their larger counterparts due to their reduced representational capacity. |
| Approach: | They use the Pythia model suite to analyse the training dynamics that underlie this phenomenon. |
| Outcome: | The proposed model suite enables us to examine the training dynamics of small models. |
Similar Papers
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining ever-larger language models on massive corpora requires enormous amounts of compute. |
| Approach: | They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance . |
| Outcome: | The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data. |
Stable Language Model Pre-training by Reducing Embedding Variability (2024.emnlp-main)
Copied to clipboard
| Challenge: | Stable pre-training is essential for achieving better-performing language models, but tracking pre-train stability is impractical due to high computational costs. |
| Approach: | They propose to use Token Embedding Variability as a proxy to estimate pre-training stability. |
| Outcome: | The proposed method improves stability and lowers perplexities even at deeper layer counts. |
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages (2025.acl-srw)
Copied to clipboard
| Challenge: | Low-resource languages (LRLs) face significant challenges in natural language processing due to limited data. |
| Approach: | They evaluate adapter-based methods for adapting mLMs to low-resource languages . they use unstructured text and structured knowledge from ConceptNet to evaluate adapters . |
| Outcome: | The proposed methods outperform large language models and LLaMA-3 and deepSeek-R1 models on low training data. |
Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies (2025.acl-long)
Copied to clipboard
| Challenge: | Existing scaling laws suggest augmenting model size and training data results in enhanced performance, but recent studies reveal deviations, particularly in large language models, where performance improvements decelerate—a phenomenon known as sub-scaling. |
| Approach: | They propose a sub-optimal scaling law that better predicts performance in sub-scaling regimes by examining data quality and training strategies. |
| Outcome: | The proposed scaling law better predicts performance in sub-scaling regimes, highlighting the importance of data quality and diversity. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |
Small Language Models Are Good Too: An Empirical Study of Zero-Shot Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Using small language models, we challenge the dominance of large models in text classification by prompting. |
| Approach: | They compare the performance of small and large language models in a zero-shot context using different architectures and scoring functions. |
| Outcome: | The proposed model outperforms large models in a zero-shot context. |
Language Models are Few-Shot Butlers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models demonstrate strong performance in most NLP tasks when fine-tuned on small task-specific datasets. |
| Approach: | They propose a two-stage procedure to learn from a small set of demonstrations and a simple reinforcement learning algorithm to improve by interacting with an environment. |
| Outcome: | The proposed method improves with only 1.2% of the demonstrations and a simple reinforcement learning algorithm over existing methods in the ALFWorld environment. |
Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models require huge training corpora, which is unobtainable for most NLP practitioners. |
| Approach: | They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget. |
| Outcome: | The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level (2021.findings-acl)
Copied to clipboard
| Challenge: | Larger models have higher out-of-distribution robustness, while smaller models have lower accuracy on rare subgroups. |
| Approach: | They develop statistically rigorous methods to investigate whether large models are better on every instance . they find that individual predictions are highly sensitive to noise in the randomness in training . |
| Outcome: | The proposed model is worse than BERT-MINI on 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%. |