Papers by Nir Yarden
Transformers are Multi-State RNNs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Schwartz et al., 2017) have been using transformers for long-range tasks for NLP since the 1990s. |
| Approach: | They propose a transformer-only transformer with unlimited hidden state size that can be converted into bounded multistate RNNs by fixing the size of their hidden state. |
| Outcome: | The proposed compression policy outperforms baseline compression policies on long range tasks and LLMs. |