| Challenge: | Term memory networks (RNNs) are difficult to optimize due to gradient vanishing and explosion. |
| Approach: | They propose a neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. |
| Outcome: | The proposed method improves state-of-the-art performance on short and long sequences and generates coherent, novel text articles with thousands of tokens. |
Similar Papers
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Improving the Transformer Translation Model with Document-Level Context (D18-1)
Copied to clipboard
| Challenge: | Existing models for document-level context translation ignore documentlevel context. |
| Approach: | They propose a document-level context encoder to represent document- level context and integrate it into the Transformer model. |
| Outcome: | Experiments on NIST Chinese-English and IWSLT French-English datasets show that the proposed translation model outperforms the Transformer model significantly. |
Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for text segmentation use supervised and unsupervised learning to perform tasks such as text summarization and keyword extraction. |
| Approach: | They propose a transformer over transformer framework to perform neural text segmentation. |
| Outcome: | The proposed framework outperforms state-of-the-art models in terms of semantic coherence measure . bottom-level sentence encoders pre-trained on specific languages yield better performance . |
Shortformer: Better Language Modeling using Shorter Inputs (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods require computationally expensive relative position embeddings. |
| Approach: | They propose two methods that decrease input length to improve perplexity and perplexability. |
| Outcome: | The proposed methods speed up training by a factor of 1.65 and reduce memory usage. |
Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that the effective use of contextual information between sentences can achieve better performance in document-level machine translation. |
| Approach: | They propose a recurrent memory unit to the Transformer to support the information exchange between the sentence and previous context. |
| Outcome: | The proposed model outperforms the previous work on TED and News by 0.91 s-BLEU and 1.49 d-BLUE on average. |
Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies . |
| Approach: | They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships. |
| Outcome: | The proposed model can be computed in parallel and improves on the WMT 14 tasks. |
Do Transformers Need Deep Long-Range Memory? (2020.acl-main)
Copied to clipboard
| Challenge: | Deep attention models have advanced the modelling of sequential data across many domains. |
| Approach: | They propose to use a Transformer augmented with a long-range memory to model sequential data across many domains. |
| Outcome: | The Transformer-XL has a long-range memory at every layer of the network, rendering its state thousands of times larger than RNN predecessors. |
Investigating Efficiently Extending Transformers for Long Input Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained Transformer models have proven capable at tackling natural language tasks, but handling long sequence inputs still poses a significant challenge. |
| Approach: | They propose an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. |
| Outcome: | The proposed model achieves strong performance on long input summarization tasks comparable with much larger models. |
Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT). |
| Approach: | They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors. |
| Outcome: | The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets. |
ResFormer: All-Time Reservoir Memory for Long Sequence Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models with quadratic time and memory complexity restrict input length . however, analyzing extensive sequential contexts is challenging . |
| Approach: | They propose a neural network architecture that captures contextual dependencies in linear time and a nonlinear readout to model short-term dependencies within sentences. |
| Outcome: | The proposed model outperforms baseline models on EmoryNLP datasets and on IEMOCAP and MultiWOZ datasets. |