Papers by Jan Chorowski
Efficient Transformers with Dynamic Token Pooling (2023.acl-long)
Copied to clipboard
| Challenge: | Hourglass Transformers is a computationally efficient model that can be used to reduce the sequence length in the intermediate layers. |
| Approach: | They propose a dynamic-pooling mechanism which predicts segment boundaries in an autoregressive fashion. |
| Outcome: | The proposed model is faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget. |