Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |
Similar Papers
Efficient Classification of Long Documents Using Transformers (2022.acl-short)
Copied to clipboard
| Challenge: | Several methods have been proposed for classifying long textual documents using Transformers, but there is a lack of consensus on a benchmark to enable a fair comparison among different approaches. |
| Approach: | They propose to use a dataset to evaluate the relative efficacy of various models for long document classification using Transformers. |
| Outcome: | The proposed models outperform simple baseline models and yield inconsistent performance across datasets. |
Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text. |
| Approach: | They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text. |
| Outcome: | The proposed models can process longer text and provide practical advice for long document classification tasks. |
Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to model long documents are difficult due to the quadratic complexity of text length. |
| Approach: | They propose a hierarchical interactive Transformer for efficient long document modeling. |
| Outcome: | Extensive experiments on three benchmark datasets validate the efficiency and effectiveness of Hi-Transformer in long document modeling. |
Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. |
| Approach: | They propose to use simple neural models and simple embeddings to improve document matching by taking significantly less training time, energy, and memory. |
| Outcome: | The proposed models outperform the more complex BERT-based models while taking significantly less training time, energy, and memory. |
On Pruning State-Space LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
CodeSSM: Towards State Space Models for Code Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformers have limitations, such as quadratic complexity and high inference costs. |
| Approach: | They propose a state space model that is trained on code corpora to assess its effectiveness. |
| Outcome: | The proposed model reduces memory usage by up to 64% compared to transformers at a context length of 2048. |
Scaling up the State Size of RNN LLMs for Long-Context Scenarios (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RNN-based LLMs struggle with long-context scenarios due to their quadratic computational complexity and linear memory requirements. |
| Approach: | They propose an efficient scaling method to scale RNN models to match the 2k context length of Transformers with small parameters overhead. |
| Outcome: | The proposed method improves long-context understanding and improves performance on FDA recall-intensive tasks. |
Document Structure in Long Document Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing long-document Transformers do not learn representations of document structure during pretraining. |
| Approach: | They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference. |
| Outcome: | The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance. |
Long-Span Summarization via Local Attention and Content Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based models are state-of-the-art for a wide range of natural language processing tasks, including document summarization. |
| Approach: | They exploit large pre-trained transformer-based models and address long-span dependencies in abstractive summarization using two methods: local self-attention; and explicit content selection. |
| Outcome: | The proposed models achieve state-of-the-art on Spotify Podcast, arXiv, and PubMed datasets. |