Papers with AdaMix
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)
Copied to clipboard
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao
| Challenge: | Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task. |
| Approach: | They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning. |
| Outcome: | The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder. |
AdaMix: Adaptive Mixing for Short and Long Reasoning Adapters (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large reasoning models have improved efficiency but still face limitations such as conflicting objectives and limited adaptability. |
| Approach: | They propose an adaptive reasoning framework that applies a uniform, computation-intensive deep reasoning strategy to all problems. |
| Outcome: | The proposed framework reduces the average response length of DeepSeek-R1-Distill-Qwen-7B by 54.9% while improving accuracy by up to 4.8% on five mathematical datasets. |