LOCOST: State-Space Models for Long Document Abstractive Summarization (2024.eacl-long)
Copied to clipboard
Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari
| Challenge: | State-space models are a low-complexity alternative to transformers for text generation . however, the quadratic complexity of the input length restricts the application of large pretrained models to long texts. |
| Approach: | They propose an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. |
| Outcome: | The proposed model saves memory and memory during training and inference time while saving 50% and 87% of memory. |
Similar Papers
Investigating Efficiently Extending Transformers for Long Input Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained Transformer models have proven capable at tackling natural language tasks, but handling long sequence inputs still poses a significant challenge. |
| Approach: | They propose an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. |
| Outcome: | The proposed model achieves strong performance on long input summarization tasks comparable with much larger models. |
Discrete Diffusion Language Model for Efficient Text Summarization (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing discrete diffusion models fail on conditional long-text generation due to incompatibility between the backbone architectures and the random noising process. |
| Approach: | They propose a semantic-aware noising process that enables Transformer backbones to handle long sequences effectively. |
| Outcome: | The proposed model outperforms existing models on three benchmark summarization datasets while achieving much faster inference speed compared to autoregressive models. |
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | We present a method to produce abstractive summaries of documents that exceed several thousand words . we compare transformer based methods to extractive methods, but extractive models score higher . |
| Approach: | They propose a method to generate abstractive summaries of documents that exceed several thousand words via neural abstractive summary. |
| Outcome: | The proposed method produces abstractive summaries of documents that exceed several thousand words . it is compared with baseline methods, state-of-the-art models and variants of the proposed method . |
Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |
Stepwise Extractive Summarization and Planning with Structured Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extractive summarization use transformers to learn the structure of long inputs. |
| Approach: | They propose encoder-centric stepwise models for extractive summarization using structured transformers – HiBERT and Extended Transformers . |
| Outcome: | The proposed models outperform previous models on CNN/DailyMail extractive summarization and Rotowire table-to-text generation. |
Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text. |
| Approach: | They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text. |
| Outcome: | The proposed models can process longer text and provide practical advice for long document classification tasks. |
Efficient Attentions for Long Document Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models that use full attentions have quadratic computational and memory complexities, and are too costly for long documents. |
| Approach: | They propose an efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. |
| Outcome: | The proposed model can process ten times more tokens than current models that use full attentions. |
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (N18-2)
Copied to clipboard
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, Nazli Goharian
| Challenge: | Existing abstractive summarization models focus on summarizing sentences and short documents. |
| Approach: | They propose a hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. |
| Outcome: | The proposed model significantly outperforms state-of-the-art models on two large-scale datasets of scientific papers. |
An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021.findings-emnlp)
Copied to clipboard
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Existing models for dialogue summarization focus on extracting the main events of short conversations, but real-world dialogues are difficult to train. |
| Approach: | They propose three strategies to deal with the lengthy input problem and locate relevant information using long dialogue datasets. |
| Outcome: | The retrieve-then-summarize pipeline models yield the best performance on three long dialogue datasets. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |