| Challenge: | Recent research suggests deep neural networks are dramatically over-parametrized. |
| Approach: | They propose that large, over-parameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better) test accuracy. |
| Outcome: | The proposed models can achieve commensurate performance using the same initialization as the original model. |
Similar Papers
The Open-World Lottery Ticket Hypothesis for OOD Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods of Out-of-Domain intent classification lack confidence in In- and Out- of-domain intents. |
| Approach: | They propose to prune overparameterized models to provide better confidence . they extend the Lottery Ticket Hypothesis to open-world scenarios . |
| Outcome: | The proposed model can be calibrated to distinguish In- and Out-of-domain intents . the model can also improve on open-world scenarios . |
Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance. |
| Approach: | They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks. |
| Outcome: | The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude . |
Robust Lottery Tickets for Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained language models contain smaller matching subnetworks that are not robust to adversarial examples. |
| Approach: | They propose a method to find robust tickets hidden in pre-trained language models by learning binary weight masks and an adversarial loss objective to guide the search. |
| Outcome: | The proposed method improves on previous work on adversarial robustness evaluation. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020.acl-main)
Copied to clipboard
| Challenge: | Current neural networks are heavily growing in depth, with many fully connected layers. |
| Approach: | They propose to combine stabilized lottery ticket pruning with unstructured pruning to improve model performance. |
| Outcome: | The proposed pruning techniques outperform all other techniques for even higher sparsity levels. |
Less Is More: Domain Adaptation with Lottery Ticket for Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation paradigms for reading comprehension require large amounts of annotation data to achieve the desired task performance. |
| Approach: | They propose a few-shot domain adaptation paradigm for reading comprehension . they introduce self-attention attribution to weigh parameters and refine the lottery subnetwork . |
| Outcome: | The proposed model outperforms the full model fine-tuning adaptation on four out of five domains with a small amount of data available for adaptation. |
Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models contain matching subnetworks that have similar transfer learning performance as the original PLM. |
| Approach: | They propose to prune matching subnetworks using magnitude-based pruning . they propose to optimize the subnetwork structure towards the pre-training objectives . |
| Outcome: | The proposed method is more efficient in searching subnetworks and advantageous when fine-tuning within a range of data scarcity. |
Analyzing the Inner Workings of Transformers in Compositional Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization abilities of neural models have focused on benchmarks, but the results do not reflect the underlying competence of the model. |
| Approach: | They propose to find an existing subnetwork that contributes to the generalization performance and perform causal analyses on how the model utilizes syntactic features. |
| Outcome: | The proposed model relies on syntactic features but the subnetwork with better generalization performance relies mainly on a non-compositional algorithm . |
When BERT Plays the Lottery, All Tickets Are Winning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large Transformer-based models are reduced to a smaller number of self-attention heads and layers. |
| Approach: | They propose to prune BERT self-attention heads and layers to find subnetworks with comparable performance . they also extend this technique to multi-layer perceptrons to find out if they are unstable . |
| Outcome: | The proposed models are able to achieve 90% of full model performance with structured pruning and similar-sized subnetworks sampled from the rest of the model perform worse. |
Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to limit overfitting of training domains are rooted in this problem . domain generalization (DG) seeks to train models on a small number of source domains . |
| Approach: | They propose to use knowledge distillation to train models on a small number of source domains to maximize their zero-shot out-of-domain utility. |
| Outcome: | The proposed model learns its source domains better and has better out-of-domain generalization . the proposed model outperforms existing approaches that aim to limit overfitting . |