Papers with FlashAttention-2
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of attention-based models for long sequences poses challenges for long-sequence modeling . attention tensorization can be used to extend context lengths with improved efficiency . tenorization enables training of LLMs with context length longer than those trained on . |
| Approach: | They propose to tensorize long input sequences into compact tenses followed by attention on each transformed dimension. |
| Outcome: | The proposed model can be used as efficient transformer backbones to extend input context length with improved memory and time efficiency. |
LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling (2025.findings-emnlp)
Copied to clipboard
Zeyu Liu, Souvik Kundu, Lianghao Jiang, Anni Li, Srikanth Ronanki, Sravan Babu Bodapati, Gourav Datta, Peter Anthony Beerel
| Challenge: | a novel linearization framework is proposed to reduce the cost of training transformers from scratch. |
| Approach: | They propose a linear attention framework that integrates pre-trained transformers into a performant linear attention architecture. |
| Outcome: | The proposed framework improves performance on mistral-7B with 1K-length sequences and BABILong benchmarks. |