EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
Similar Papers
LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) reach hundreds of billions of parameters and require resources for training and inference stages. |
| Approach: | They propose a low-rank adapter to reduce the number of trainable parameters in a model and reduce memory requirements. |
| Outcome: | The proposed approach reduces memory and compute requirements while preserving performance. |
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent transformer language models achieve outstanding results on many downstream tasks, but their enormous size often makes them impractical on memory-constrained devices. |
| Approach: | They propose an offline compression approach that reduces the complexity of the model by enabling collaboration between modules. |
| Outcome: | The proposed approach outperforms commonly used factorization-based offline compression methods on various NLP tasks. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are sensitive to architecture and hyperparameter settings. |
| Approach: | They incorporate architecture search into a single training run through auto-sizing . they show that auto-size can improve BLEU scores by up to 3.9 points . |
| Outcome: | The proposed algorithm improves BLEU scores on low-resource language pairs while removing one-third of the parameters from the model. |
Optimizing Transformer for Low-Resource Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Language pairs with limited amounts of parallel data remain a challenge for neural machine translation. |
| Approach: | They propose to optimize a Transformer model for low-resource conditions to improve translation quality by 7.3 BLEU points compared to the default settings. |
| Outcome: | The proposed model improves translation quality up to 7.3 BLEU points compared to the default settings on the IWSLT14 training data compared with the Transformer model. |
Star-Transformer (N19-1)
Copied to clipboard
| Challenge: | Existing models with fully-connected attention connections are heavy and require large training data. |
| Approach: | They propose a lightweight alternative to the Transformer by sparsifying the fully-connected structure with a star-shaped topology. |
| Outcome: | The proposed model achieves significant performance improvements on 22 datasets on four tasks. |
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to character-level language modeling have suffered from high learning complexity caused by inherently long character sequences. |
| Approach: | They propose a method that efficiently reduces the computational cost and parameter size of Transformer by splitting feature space into multiple groups, factorizing the calculation paths, and reducing computations for the group interaction. |
| Outcome: | The proposed model reduces the computational cost and parameter size of Transformer on two benchmark tasks, enwik8 and text8, and it performs well. |
Hierarchical Transformers Are More Efficient Language Models (2022.findings-naacl)
Copied to clipboard
Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, Henryk Michalewski
| Challenge: | Transformers are impressive but inefficient and costly, which limits their applications and accessibility. |
| Approach: | They first use different ways to downsample and upsamplify activations in Transformers to make them hierarchical. |
| Outcome: | The proposed model outperforms Transformers on the ImageNet32 and enwik8 benchmarks. |
Approximating Two-Layer Feedforward Networks for Efficient Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work uses sparse Mixtures of Experts (MoEs) to build resource-efficient large language models. |
| Approach: | They propose a general framework that unifies various methods to build two-layer NNs . they propose methods to improve both MoEs and PKMs based on their results . |
| Outcome: | The proposed framework improves both MoEs and product-key memories (PKMs) it shows that MoE's are competitive with dense Transformer-XL on two different scales while being much more resource efficient. |
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |