ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks. |
| Approach: | They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy. |
| Outcome: | The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks. |
Similar Papers
Training-free Neural Architecture Search for RNNs and Transformers (2023.acl-long)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures. |
| Approach: | They develop a new NAS metric that predicts the trained performance of an RNN architecture and significantly outperforms existing NAS metrics. |
| Outcome: | The proposed metric outperforms existing training-free metrics on the NAS-Bench-NLP benchmark. |
TRAMS: Training-free Memory Selection for Long-range Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods like Transformer-XL are plagued by ineffective memory selections due to the high number of tokens involved in attention calculation. |
| Approach: | They propose a plug-and-play strategy that selects tokens participating in attention calculation based on one simple metric and ignores the other ones. |
| Outcome: | The proposed strategy keeps tokens with high attention scores and ignores the other ones on word-level and character-level benchmarks without additional training or adding additional parameters. |
Life after BERT: What do Other Muppets Understand about Language? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained transformer analysis studies focus on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. |
| Approach: | They utilize oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. |
| Outcome: | The proposed model fails to resolve compositional questions in a zero-shot fashion, suggesting that pre-training objectives are not predictive of a model’s linguistic capabilities. |
Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are sensitive to architecture and hyperparameter settings. |
| Approach: | They incorporate architecture search into a single training run through auto-sizing . they show that auto-size can improve BLEU scores by up to 3.9 points . |
| Outcome: | The proposed algorithm improves BLEU scores on low-resource language pairs while removing one-third of the parameters from the model. |
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers (2023.acl-long)
Copied to clipboard
Linyuan Gong, Chenyan Xiong, Xiaodong Liu, Payal Bajaj, Yiqing Xie, Alvin Cheung, Jianfeng Gao, Xia Song
| Challenge: | Recent work in NLP has shown that pretrained language models have made notable progress toward generalization to unseen tasks. |
| Approach: | They propose to pretrain T5 using an auxiliary model to construct more challenging token replacements for the main model to denoise. |
| Outcome: | The proposed model outperforms similar-sized baseline models on prompted NLP benchmarks and rivals the state-of-the-art model with only **8%** of its parameters. |
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Sparse attention is a promising strategy to extend long-context capabilities in LLMs . but its efficiency–accuracy trade-offs remain unclear due to the lack of comprehensive evaluation . |
| Approach: | They evaluate sparse attention methods across multiple model families and sizes . they find larger sparser models outperform smaller dense ones at equivalent cost . |
| Outcome: | The proposed methods outperform smaller sparse models at equivalent cost and improve the Pareto frontier. |
Effective Pretraining Objectives for Transformer-based Autoencoders (2022.findings-emnlp)
Copied to clipboard
| Challenge: | ELECTRA is more accurate than BERT, but it is not clear if this is due to its innovative architecture or to the long and extensive training, which highly increases the computation cost for obtaining the final language model. |
| Approach: | They propose to replace BERT’s Masked Language Modeling objective (MLM) with Token Detection (TD) by using a statistical approach to generate light tokens. |
| Outcome: | The proposed method can replace ELECTRA's computationally heavy generators without a significant drop in performance. |
Transformer-specific Interpretability (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque. |
| Approach: | This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions . |
| Outcome: | This tutorial aims to show how a new trending approach can be applied to Transformer-based models. |
EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments. |
| Approach: | They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models. |
| Outcome: | The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset. |
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of natural language processing models are over-parameterized and overfitted. |
| Approach: | They propose an approach to integrate dropout techniques into the training of Transformer models. |
| Outcome: | The proposed approach can achieve 1.5 BLEU improvement on IWSLT14 translation tasks and better accuracy for the classification even using strong pre-trained RoBERTa as backbone. |