Papers with SNIP
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Spectral-normalized identity priors (SNIP) is a structured pruning approach for a Transformer model. |
| Approach: | They propose a structured pruning approach which penalizes an entire residual module toward an identity mapping. |
| Outcome: | The proposed method improves on 5 GLUE benchmark tasks while maintaining comparable performance. |
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrain-finetuned models are increasingly complex and require more parameters to match the performance of full fine-tuning. |
| Approach: | They propose an efficient Adapter Tuning technique that freezes pretrained language models and fine-tunes a few extra modules. |
| Outcome: | The proposed setting outperforms the standard Adapter Tuning by 80% . the proposed setting is easy to use and has a high sparse ratio . |