| Challenge: | Pre-trained language models are computationally expensive to fine-tune and require large storage. |
| Approach: | They propose a method to identify the influence of each adapter module and a way to prune adapters based on the Lottery Ticket Hypothesis. |
| Outcome: | The proposed model reduces size significantly while keeping performance intact. |
Similar Papers
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020.acl-main)
Copied to clipboard
| Challenge: | Current neural networks are heavily growing in depth, with many fully connected layers. |
| Approach: | They propose to combine stabilized lottery ticket pruning with unstructured pruning to improve model performance. |
| Outcome: | The proposed pruning techniques outperform all other techniques for even higher sparsity levels. |
Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that attention heads are not confident in their decisions and can be pruned. |
| Approach: | They apply the lottery ticket hypothesis to prune heads in early training . they find that the pruned model is 1.5 times faster at inference . |
| Outcome: | The proposed method is 1.5 times faster at inference, but at the cost of longer training. |
AdapterDrop: On the Efficiency of Adapters in Transformers (2021.emnlp-main)
Copied to clipboard
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, Iryna Gurevych
| Challenge: | Recent approaches to transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. |
| Approach: | They propose a method to remove adapters from transformer layers during training and inference . they show that AdapterDrop can dynamically reduce computational overhead . |
| Outcome: | The proposed approach reduces computational overhead while maintaining performance over multiple tasks with minimal loss of performance. |
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models contain matching subnetworks that have similar transfer learning performance as the original PLM. |
| Approach: | They propose to prune matching subnetworks using magnitude-based pruning . they propose to optimize the subnetwork structure towards the pre-training objectives . |
| Outcome: | The proposed method is more efficient in searching subnetworks and advantageous when fine-tuning within a range of data scarcity. |
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance. |
| Approach: | They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands. |
| Outcome: | Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model. |
Adapter Pruning using Tropical Characterization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on adapter pruning have not examined the optimal number of adapter parameters needed for downstream applications. |
| Approach: | They propose an adapter pruning approach that prunes adapter parameters without changing the orientation of underlying tropical hypersurfaces. |
| Outcome: | The proposed approach prunes adapter layers without changing the orientation of underlying tropical hypersurfaces. |
Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pruning results on benchmark transformers, such as BERT, are not as remarkable as those of convolutional neural networks. |
| Approach: | They propose to apply a knowledge-aware pruning process to transformer-based pre-trained language models to reduce model size and model weight. |
| Outcome: | The proposed pruning method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy. |
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating knowledge from multiple tasks suffer from catastrophic forgetting and difficulties in dataset balancing. |
| Approach: | They propose an algorithm that extracts and combine adapters in a knowledge composition step. |
| Outcome: | The proposed class outperforms traditional methods such as full fine-tuning and multi-task learning on 16 diverse NLU tasks. |
Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Multilingual transformer-based models demonstrate remarkable zero and few-shot transfer across languages by learning and reusing language-agnostic features. |
| Approach: | They propose to prune attention heads from a fixed model to reduce interference as a model acquires more languages. |
| Outcome: | The proposed model can reduce interference by pruning language-specific attention heads. |
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |