Instability in Downstream Task Performance During LLM Pretraining (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models shows that task scores fluctuate throughout training . |
| Approach: | They empirically analyze the stability of downstream task performance in an LLM . |
| Outcome: | The proposed methods improve performance stability without changes to the training procedure. |
Similar Papers
Harnessing Consistency for Robust Test-Time LLM Ensemble (2026.findings-eacl)
Copied to clipboard
Zhichen Zeng, Qi Yu, Xiao Lin, Ruizhong Qiu, Xuying Ning, Tianxin Wei, Yuchen Yan, Jingrui He, Hanghang Tong
| Challenge: | Existing efforts to improve LLM ensemble quality have focused on model consistency, but failures are often due to heterogeneous tokenization schemes and varying model expertise. |
| Approach: | They propose a plug-and-play technique that harnesses model consistency for robust LLM ensemble. |
| Outcome: | The proposed technique improves ensemble performance and robustness against erroneous signals. |
Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning large language models for downstream tasks often leads to catastrophic forgetting, notably degrading the safety of original alignments. |
| Approach: | They propose to merge the weights of pre- and post-fine-tuned models to improve safety while enhancing performance. |
| Outcome: | Experiments across different downstream tasks and models validate the method’s practicality and effectiveness. |
Practical Guidelines for Model Merging in LLMs Pre-Training (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on model merging have focused on stable learning rate regimes, but its effectiveness during LLM pre-training remains underexplored. |
| Approach: | They systematically investigate model merging across training phases, focusing on the transition from stable to decaying learning rates. |
| Outcome: | The proposed methods improve performance during stable learning rate regimes but diminish under decay, a phe-nomenon that is linked to reduced checkpoint diversity and lower parameter-space variability. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
Imbalanced Gradients in RL Post-Training of Multi-Task LLMs (2026.findings-eacl)
Copied to clipboard
Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni
| Challenge: | Large-gradient tasks can achieve similar or even much lower learning gains than small-grading ones. |
| Approach: | They show that large-gradient tasks can achieve lower learning gains than small-grading ones . large-grade tasks can accomplish similar or even lower learning gain than small grade ones if they are large . |
| Outcome: | The proposed approach fails when certain tasks produce larger gradients . Large-gradient tasks can achieve lower learning gains than small-gradent ones . |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Efficient Domain Continual pretraining by Mitigating the Stability Gap (2025.acl-long)
Copied to clipboard
| Challenge: | Continual pretraining is an important approach for Large Language Models to improve their performance in target domains, learn new topics and languages, and even boost their general capabilities. |
| Approach: | They propose a training strategy that mitigates instability by increasing the number of epochs, along with two data sampling strategies targeting data domain relevance and corpus distribution. |
| Outcome: | The proposed training strategy improves the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% using only 40% of the original training budget, while also enhancing general task performance without causing forgetting. |
Probing the Emergence of Cross-lingual Alignment during LLM Training (2024.findings-acl)
Copied to clipboard
| Challenge: | Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. |
| Approach: | They propose that LLMs can align languages without explicit supervision from parallel sentences without a single linguistic feature. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer even when the vocabularies of two languages have a null intersection, i.e., no tokens are shared. |
Tokenization is Sensitive to Language Variation (2025.findings-acl)
Copied to clipboard
| Challenge: | Variation in language is often linked to regional, social, and contextual factors. |
| Approach: | They propose a method to estimate tokenizer impact on downstream LLM performance . they pre-train BERT models with the popular Byte-Pair Encoding algorithm . |
| Outcome: | The proposed model improves on Rényi efficiency and other metrics on language variation. |
Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies on self-training report seemingly contradictory outcomes. |
| Approach: | They use OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with different prompting strategies and data filtering. |
| Outcome: | The proposed model collapse is inherent to the training procedure itself, while self-improvement is likely owes its success to human-designed, strategic synthetic pipelines that inject external intelligence. |