Papers by Leandros Tassiulas
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of attention-based models for long sequences poses challenges for long-sequence modeling . attention tensorization can be used to extend context lengths with improved efficiency . tenorization enables training of LLMs with context length longer than those trained on . |
| Approach: | They propose to tensorize long input sequences into compact tenses followed by attention on each transformed dimension. |
| Outcome: | The proposed model can be used as efficient transformer backbones to extend input context length with improved memory and time efficiency. |