| Challenge: | Existing approaches to model long-range dependencies in text are limited to 512 tokens . however, the amount of compute in attention depends quadratically on the number of tokens in an input text passage. |
| Approach: | They propose a technique that summarises text into a memory table to be used in a second read of the text. |
| Outcome: | The proposed method outperforms models of comparable size on several question answering datasets and sets a new state of the art on the NarrativeQA task, with questions about entire books. |
Similar Papers
NarrativeXL: a Large-scale Dataset for Long-Term Memory Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 990,595 questions are needed to solve ultra-long-context reading comprehension problems. |
| Approach: | They propose a large-scale reading comprehension dataset using 1,500 hand-curated fiction books and a set of reading comprehension questions based on these summaries. |
| Outcome: | The proposed reading comprehension dataset is larger than the closest alternatives and has more questions than the existing models. |
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine reading comprehension (MRC) on long texts typically chunk text into equally-spaced segments without considering information from other segments. |
| Approach: | They propose to let a model learn to chunk in a more flexible way via reinforcement learning. |
| Outcome: | The proposed model extracts a text span from document and query as answer . previous models can only take a fixed-length (e.g., 512) text as input . |
RoR: Read-over-Read for Long Document Machine Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension are limited to individual chunks due to encoding length constraint. |
| Approach: | They propose a read-over-read method that expands the reading field from chunk to document by predicting regional answers for each chunk. |
| Outcome: | Extensive experiments on QuAC and TriviaQA show that the proposed model performs well for long document reading. |
MemoReader: Large-Scale Reading Comprehension through Neural Memory Controller (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to machine reading comprehension are limited in understanding, up to a few paragraphs, failing to comprehend lengthy documents. |
| Approach: | They propose a deep neural network architecture to handle a long-range dependency in RC tasks. |
| Outcome: | The proposed method outperforms existing methods especially for lengthy documents. |
Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Data (P19-1)
Copied to clipboard
| Challenge: | Existing QA methods lack scalability and performance is difficult to solve with document-level contexts. |
| Approach: | They propose an end-to-end deep network model that sequentially reads the input contexts into an external memory while replacing memories that are less important for answering unseen questions. |
| Outcome: | The proposed model improves on a synthetic dataset and real-world large-scale textual and video QA datasets. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |
TRAMS: Training-free Memory Selection for Long-range Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods like Transformer-XL are plagued by ineffective memory selections due to the high number of tokens involved in attention calculation. |
| Approach: | They propose a plug-and-play strategy that selects tokens participating in attention calculation based on one simple metric and ignores the other ones. |
| Outcome: | The proposed strategy keeps tokens with high attention scores and ignores the other ones on word-level and character-level benchmarks without additional training or adding additional parameters. |
Book QA: Stories of Challenges and Opportunities (D19-58)
Copied to clipboard
| Challenge: | Existing approaches to answer questions based on the full text of books are limited by their unique characteristics. |
| Approach: | They propose a system for answering questions based on the full text of books . they use a memory network to reason and predict an answer, and a novel question generator to improve generalization. |
| Outcome: | The proposed system improves on the recently published NarrativeQA corpus on Who questions . it shows that the proposed system is highly challenging and needs more research . |
Proceedings of the 2nd Workshop on Machine Reading for Question Answering (D19-58)
Copied to clipboard
| Challenge: | a workshop focuses on machine reading for question answering . despite recent progress, there is much to be desired about these datasets and systems . |
| Approach: | This year, they present a shared task on machine reading for question answering . they adapt and unified 18 distinct question answering datasets into the same format . |
| Outcome: | The proposed system achieves an average F1 score of 72.5 on the held-out datasets. |
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on parametric models that store knowledge in parameters or retrieval-augmented models that have access to external knowledge sources. |
| Approach: | They propose a parametric parametric model that stores knowledge in its parameters or a retrieval-augmented model that has access to external knowledge sources. |
| Outcome: | The proposed method runs substantially faster across the board and produces more accurate results on WoW and ELI5. |