ChuLo: Chunk-Level Key Information Representation for Long Document Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Traditional approaches to truncate inputs, sparse self-attention, and chunking often lead to information loss and hinder the model’s ability to capture long-range dependencies. |
| Approach: | They propose a novel chunk representation method that uses unsupervised keyphrase extraction to group input tokens to retain core document content while reducing input length. |
| Outcome: | The proposed method minimizes information loss and improves the efficiency of Transformer-based models. |
Similar Papers
Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Current methods for improving large language models rely on splitting long contexts into fixed-length chunks, compromising accuracy. |
| Approach: | They propose a method for dynamically separating and selecting chunks of long context, facilitating a more streamlined input for LLMs. |
| Outcome: | The proposed approach outperforms baseline methods on single-hop and multi-hop question-answering benchmarks. |
Efficient Long-Text Understanding with Short-Text Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing transformer-based pretrained language models cannot be applied to long sequences due to their quadratic complexity. |
| Approach: | They propose a simple approach to long sequences that re-uses battle-tested short-text pretrained LMs. |
| Outcome: | The proposed approach is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. |
Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text. |
| Approach: | They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text. |
| Outcome: | The proposed models can process longer text and provide practical advice for long document classification tasks. |
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine reading comprehension (MRC) on long texts typically chunk text into equally-spaced segments without considering information from other segments. |
| Approach: | They propose to let a model learn to chunk in a more flexible way via reinforcement learning. |
| Outcome: | The proposed model extracts a text span from document and query as answer . previous models can only take a fixed-length (e.g., 512) text as input . |
Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers (2024.acl-long)
Copied to clipboard
| Challenge: | Existing transformer-based models struggle with long-sequence processing due to computational costs . a framework to enhance long-content processing of transformers is proposed . |
| Approach: | They propose a framework to enhance long-sequence processing of transformers by three steps . they demonstrate that the framework significantly outperforms prior long-quence processors . |
| Outcome: | The proposed framework outperforms baseline models on long-sequence summarization and reading comprehension tasks. |
RoR: Read-over-Read for Long Document Machine Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension are limited to individual chunks due to encoding length constraint. |
| Approach: | They propose a read-over-read method that expands the reading field from chunk to document by predicting regional answers for each chunk. |
| Outcome: | Extensive experiments on QuAC and TriviaQA show that the proposed model performs well for long document reading. |
CoLT5: Faster Long-Range Transformers with Conditional Computation (2023.emnlp-main)
Copied to clipboard
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
| Challenge: | Many natural language processing tasks require long inputs, but processing long documents with a Transformer model is expensive due to quadratic attention complexity and applying feedforward and attention projection layers to every input token. |
| Approach: | They propose a long-input Transformer model that builds on the intuition that some tokens are more important than others and uses conditional computation to devote more computation to important tokens. |
| Outcome: | The proposed model achieves stronger performance than LongT5 with faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. |
Document Structure in Long Document Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing long-document Transformers do not learn representations of document structure during pretraining. |
| Approach: | They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference. |
| Outcome: | The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance. |
Long Document Summarization with Top-down and Bottom-up Inference (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent models infer latent representations of words or tokens with a transformer encoder, which is bottom-up and thus does not capture long-distance context well. |
| Approach: | They propose a method to infer latent representations of words or tokens in documents . they assume a hierarchical structure of a document where top-level captures long range dependency . |
| Outcome: | The proposed model can summarize an entire book and achieve competitive performance on a wide range of document summarization benchmarks. |
IRIS: Interpretable Retrieval-Augmented Classification for Long Interspersed Document Sequences (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models for document classification struggle with long-text processing due to quadratic computational complexity in the self-attention module. |
| Approach: | They propose a framework that utilizes retrieval to efficiently classify long documents . they use a quadratic attention matrix to capture dependencies between tokens in an input sequence . |
| Outcome: | The proposed framework excels in clinical note disease risk prediction tasks . it can process arbitrarily long documents without increasing computational cost and trainable on a single GPU. |