Learning Architectures from an Extended Search Space for Language Modeling (2020.acl-main)
Copied to clipboard
Yinqiao Li, Chi Hu, Yuhao Zhang, Nuo Xu, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, Changliang Li
| Challenge: | Neural architecture search (NAS) has advanced in recent years but most NAS systems restrict search to learning architectures of a recurrent or convolutional cell. |
| Approach: | They propose a general approach to learn both intra-cell and inter-cell architectures . they implement their approach in a differentiable architecture search system . |
| Outcome: | The proposed approach outperforms the baseline on PTB and WikiText data and shows good transferability to other systems. |
Similar Papers
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition (D19-1)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) is a popular approach for finding new models and freeing researchers from the hard work of designing network architectures. |
| Approach: | They propose differentiable neural architecture search methods for natural language processing . they remove the softmax-local constraint and apply it to named entity recognition . |
| Outcome: | The proposed method outperforms strong baselines on the language modeling task. |
FENAS: Flexible and Expressive Neural Architecture Search (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to architecture search have shown good improvements in terms of performance with reasonable training speed. |
| Approach: | They propose an algorithm with more activation functions, input edges, and atomic operations to search for architectures that are optimal for given task. |
| Outcome: | The proposed algorithm reproduces well-known LSTM and GRU architectures and initializes with them for finding architectures more efficiently. |
Continual and Multi-Task Architecture Search (P19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that architecture search can improve performance on language modeling and image classification tasks with reasonable training speed. |
| Approach: | They propose a continual architecture search approach that continually evolves the model parameters during sequential training of several tasks without losing performance on previously learned tasks. |
| Outcome: | The proposed approach improves language modeling and image classification with reasonable training speed and a weight-sharing strategy. |
LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding (2025.emnlp-main)
Copied to clipboard
Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li
| Challenge: | Recent advances in Large Language Models have opened new avenues for solving complex optimization problems, including Neural Architecture Search (NAS). |
| Approach: | They propose a framework that leverages LLMs for cross-domain neural architecture optimization without extensive domain-specific tuning. |
| Outcome: | The proposed framework achieves competitive performance in both in-domain and out-of-domain tasks. |
RankNAS: Efficient Neural Architecture Search by Pairwise Ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods require training millions of architectures to estimate the accuracy of the search results. |
| Approach: | They propose a performance ranking method (RankNAS) that uses pairwise ranking and search space pruning to enlarge the search space. |
| Outcome: | The proposed method significantly accelerates NAS through pairwise ranking and search space pruning. |
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Parameter-efficient tuning (PET) methods fit pre-trained language models to downstream tasks by computing a small compressed update for a subset of parameter updates or appending and fine-tuning a few new parameters to the pre-train network. |
| Approach: | They propose a parameter-efficient tuning method that prunes PET architectures using structured pruning. |
| Outcome: | The proposed method performs well in practice but could be improved via NAS . it can be used to fine-tune large pre-trained language models on multiple tasks . |
Discovering Better Model Architectures for Medical Query Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has attracted intense attention in computer vision and NLP. |
| Approach: | They propose to use neural architecture search to optimize model architectures for medical questions . they propose to modify the ENAS method to accelerate and stabilize the search results . |
| Outcome: | The proposed approach outperforms baseline models on two medical questions . it is compared with other NAS methods and shows that it provides the best results . |
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved great success in natural language processing. |
| Approach: | They propose a method that automatically searches architecture hyper-parameters in BERT . they use one-shot learning and the search space to provide an adaptive development way . |
| Outcome: | The proposed method outperforms both the baseline and distillation-based methods on GLUE and SQUAD benchmarks. |
Training-free Neural Architecture Search for RNNs and Transformers (2023.acl-long)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures. |
| Approach: | They develop a new NAS metric that predicts the trained performance of an RNN architecture and significantly outperforms existing NAS metrics. |
| Outcome: | The proposed metric outperforms existing training-free metrics on the NAS-Bench-NLP benchmark. |
AutoRC: Improving BERT Based Relation Classification Models via Architecture Search (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing literature on relation classification models shows no consensus on optimal architecture . |
| Approach: | They propose a search space for BERT based relation classification models and employ an ENAS method to find better architectures. |
| Outcome: | The proposed method can find better architectures than baseline models on eight benchmark RC tasks. |