| Challenge: | Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. |
| Approach: | They propose a Fitting Paradigm and a Transfer Paradigme to investigate fit and transfer . they propose scalability and elucidate the reasons why module-wise parameter tuning underperforms . |
| Outcome: | The proposed model reduces the search complexity by reducing the search cost by lowering the search factor. |
Similar Papers
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning methods require additional measures after pre-training and before fine-uning. |
| Approach: | They propose to take parameter-efficient fine-tuning into consideration after pre-training and before fine-uning and use meta-learning to prime a model specifically for parameter-efficiency. |
| Outcome: | The proposed method improves on a pre-trained model with certain modifications and achieves 4.96 points on cross-lingual NER fine-tuning. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for reducing computational costs during pre-training have been studied, but they often degrade performance under fair conditions. |
| Approach: | They propose a method that combines parameter-efficient tuning and staged training to reduce memory requirements while maintaining comparable performance. |
| Outcome: | The proposed method reduces memory requirements by 40.4% while maintaining comparable performance. |
Parameter-Efficient Fine-Tuning: Is There An Optimal Subset of Parameters to Tune? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has illuminated the possibility of selective parameter-efficient fine-tuning, which retains the inference speed of the original model and comes at no additional computational cost. |
| Approach: | They propose to selectively update only a small subset of parameters during the fine-tuning process, keeping the remaining parameters frozen during training. |
| Outcome: | The proposed methods retain the inference speed of the original model and come at no additional computational cost. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune deep pretrained language models face catastrophic forgetting problems. |
| Approach: | They propose a recall and learn mechanism which integrates pretraining and downstream tasks into a single mechanism. |
| Outcome: | The proposed method achieves state-of-the-art performance on the GLUE benchmark and better average performance than directly fine-tuning of BERT-large. |
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | In the recent years, pre-trained language models have achieved great success in the NLP community. |
| Approach: | They propose two general strategies and an experimental procedure to troubleshoot HPO’s failure cases. |
| Outcome: | The proposed methods outperform grid search on two state-of-the-art language models using the same time budget and overfitting. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent LLM development trends involve pre-training models with a vast number of parameters on massive datasets. |
| Approach: | They propose a method that integrates parameter-efficient tuning techniques with model growth to reduce memory requirements while maintaining equivalent performance. |
| Outcome: | The proposed method reduces memory requirements by 53.9% while maintaining equivalent performance to vanilla pre-trained models on downstream tasks. |
Hop, skip, jump to Convergence: Dynamics of Learning Rate Transitions for Improved Training of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep neural networks have achieved stateof-the-art performance across a wide range of machine learning tasks. |
| Approach: | They propose to switch the learning rate at a predetermined time during training to improve the performance of large language models. |
| Outcome: | The proposed model shows that switching the learning rate causes the loss curves to contract towards each other. |