Papers by Sahaj Agarwal
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)
Copied to clipboard
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao
| Challenge: | Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task. |
| Approach: | They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning. |
| Outcome: | The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder. |