Papers with 1.3B
LEMON: Reviving Stronger and Smaller LMs from Larger LMs with Linear Parameter Fusion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train a stronger and smaller model with the help of large models are limited by the model size and performance. |
| Approach: | They propose to learn competent initial points for smaller models by fusing parameters from larger models and introduce controllable receptive fields to model prior parameter characteristics. |
| Outcome: | The proposed method outperforms baselines in terms of effectiveness and efficiency. |
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data (2025.findings-acl)
Copied to clipboard
Xuemiao Zhang, Xu Liangyu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, Xunliang Cai
| Challenge: | Existing methods of uniformly sampling data throughout the pretraining process are suboptimal because they overlook the model's evolving data preferences. |
| Approach: | They propose a Perplexity Difference (PD) based Preference Curriculum learning framework which perceives and uses the data preferred by LLMs as their capabilities improve . they propose PDPC to complete the arrangement of the dataset offline and ensure continuous training without interruption. |
| Outcome: | The proposed framework surpasses baselines on 1.3B and 3B models and achieves an increased average accuracy of over 8.1% across MMLU and CMMLU. |