Papers with trainability
ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Transformer Architecture Search methods are limited to computer vision and natural language processing tasks. |
| Approach: | They propose a Transformer Architecture Search proxy that measures trainability and expressivity of Transformer networks separately and integrates it into an effective regularized evolution framework to demonstrate its efficacy. |
| Outcome: | The proposed proxy can achieve higher correlation with the true performance of Transformer networks on computer vision and natural language processing tasks. |