Training-free Neural Architecture Search for RNNs and Transformers

Challenge:	Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures.
Approach:	They develop a new NAS metric that predicts the trained performance of an RNN architecture and significantly outperforms existing NAS metrics.
Outcome:	The proposed metric outperforms existing training-free metrics on the NAS-Bench-NLP benchmark.

Similar Papers

Overview

Challenge

Approach

Outcome

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking (2021.emnlp-main)

Copied to clipboard

Chi Hu, Chenglong Wang, Xiangnan Ma, Xia Meng, Yinqiao Li, Tong Xiao, Jingbo Zhu, Changliang Li

Green & Sustainable NLP

Challenge:	Existing methods require training millions of architectures to estimate the accuracy of the search results.
Approach:	They propose a performance ranking method (RankNAS) that uses pairwise ranking and search space pruning to enlarge the search space.
Outcome:	The proposed method significantly accelerates NAS through pairwise ranking and search space pruning.

ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)

Copied to clipboard

Jiechao Yang, Yong Liu

Semantic Text Processing

Challenge:	Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks.
Approach:	They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy.
Outcome:	The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks.

EFTNAS: Searching for Efficient Language Models in First-Order Weight-Reordered Super-Networks (2024.lrec-main)

Copied to clipboard

Juan Pablo Munoz, Yi Zheng, Nilesh Jain

Semantic Text Processing

Challenge:	Depending on the size of transformer-based models, they can be restricted from deployment in resource-constrained environments.
Approach:	They propose to combine neural architecture search and network pruning techniques to generate and train weight-sharing super-networks that contain efficient transformer-based models.
Outcome:	The proposed model achieves high-performing, high-performance subnetworks on the general language understanding evaluation and the Stanford Question Answering Dataset.

Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (2023.findings-acl)

Copied to clipboard

Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, Greg Ver Steeg

Semantic Text Processing

Challenge:	Parameter-efficient tuning (PET) methods fit pre-trained language models to downstream tasks by computing a small compressed update for a subset of parameter updates or appending and fine-tuning a few new parameters to the pre-train network.
Approach:	They propose a parameter-efficient tuning method that prunes PET architectures using structured pruning.
Outcome:	The proposed method performs well in practice but could be improved via NAS . it can be used to fine-tune large pre-trained language models on multiple tasks .

Learning Architectures from an Extended Search Space for Language Modeling (2020.acl-main)

Copied to clipboard

Yinqiao Li, Chi Hu, Yuhao Zhang, Nuo Xu, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, Changliang Li

Language Models

Challenge:	Neural architecture search (NAS) has advanced in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell.
Approach:	They propose a general approach to learn both intra-cell and inter-cell architectures . they implement their approach in a differentiable architecture search system .
Outcome:	The proposed approach outperforms the baseline on PTB and WikiText data and shows good transferability to other systems.

How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures (P18-1)

Copied to clipboard

Tobias Domhan

Semantic Text Processing

Challenge:	Neural Machine Translation (NMT) has been replaced by convolutional or self-attentional approaches.
Approach:	They propose an architecture definition language that allows for a flexible combination of common building blocks.
Outcome:	The proposed architectures can bring recurrent and convolutional models close to the Transformer architecture, but not using self-attention.

Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (D19-56)

Copied to clipboard

Kenton Murray, Jeffery Kinnison, Toan Q. Nguyen, Walter Scheirer, David Chiang

Responsible & Trustworthy NLP

Challenge:	Neural sequence-to-sequence models are sensitive to architecture and hyperparameter settings.
Approach:	They incorporate architecture search into a single training run through auto-sizing . they show that auto-size can improve BLEU scores by up to 3.9 points .
Outcome:	The proposed algorithm improves BLEU scores on low-resource language pairs while removing one-third of the parameters from the model.

Transformers are Multi-State RNNs (2024.emnlp-main)

Copied to clipboard

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, Roy Schwartz

Language Models

Challenge:	Schwartz et al., 2017) have been using transformers for long-range tasks for NLP since the 1990s.
Approach:	They propose a transformer-only transformer with unlimited hidden state size that can be converted into bounded multistate RNNs by fixing the size of their hidden state.
Outcome:	The proposed compression policy outperforms baseline compression policies on long range tasks and LLMs.

ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer (2022.acl-long)

Copied to clipboard

Ningning Wang, Guobing Gan, Peng Zhang, Shuai Zhang, Junqiu Wei, Qun Liu, Xin Jiang

Language Models

Challenge:	Existing sparse attention methods use fixed patterns to select words without considering similarities between words.
Approach:	They propose a neural clustering method which integrates into the Self-Attention Mechanism in Transformer and integrates it into the target task.
Outcome:	The proposed method outperforms two typical sparse attention methods on translation, text classification, and text matching tasks while having a comparable or even better time and memory efficiency.

TRAMS: Training-free Memory Selection for Long-range Language Modeling (2023.findings-emnlp)

Copied to clipboard

Haofei Yu, Cunxiang Wang, Yue Zhang, Wei Bi

Language Models

Challenge:	Existing methods like Transformer-XL are plagued by ineffective memory selections due to the high number of tokens involved in attention calculation.
Approach:	They propose a plug-and-play strategy that selects tokens participating in attention calculation based on one simple metric and ignores the other ones.
Outcome:	The proposed strategy keeps tokens with high attention scores and ignores the other ones on word-level and character-level benchmarks without additional training or adding additional parameters.