| Challenge: | Existing methods to deal with long-range data processing are implicit convolutions and regularized parameterization. |
| Approach: | They propose a new layer where dynamic (i.e., input-dependent) IIR filters are used to process the input sequence prior to applying conventional attention. |
| Outcome: | The proposed layer performs on-par with state-of-the-art networks with a fraction of their parameters and time complexity that is sub-quadratic with input size. |
Similar Papers
Surprisingly Easy Hard-Attention for Sequence to Sequence Learning (D18-1)
Copied to clipboard
| Challenge: | Existing attention mechanisms are hard and hard, but they are more accurate when trained. |
| Approach: | They propose to use a beam approximation of the joint distribution between attention and output to train sequence to sequence learning. |
| Outcome: | The proposed method is compared to existing attention mechanisms on five translation tasks and shows consistent gains on the same tasks. |
Chain and Causal Attention for Efficient Entity Tracking (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handle entity tracking require at least log2 (n+1) layers to handle n state changes. |
| Approach: | They propose an efficient enhancement to the standard attention mechanism to handle long-term dependencies with a single layer. |
| Outcome: | The proposed model can handle entity tracking with n state changes with a single layer. |
DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large reasoning models generate long chains of intermediate steps, but their inference cost is dominated by decoding, where each new token must attend to the entire growing sequence. |
| Approach: | They propose a training-free sparse attention mechanism that reduces inference cost by evicting entries from the key-value cache. |
| Outcome: | The proposed model matches or surpasses full attention on reasoning benchmarks . it reduces the number of attended tokens by up to 4.25 and delivers 1.54 speedup . |
Focus-Constrained Attention Mechanism for CVAE-based Response Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models generate high-frequency but trivial responses such as "I don't know" or "I'm ok" due to the discrepancy in discourse-level information, standard models generate one-to-many relationships. |
| Approach: | They propose to transform coarse-grained discourse-level information into fine-grounded word-level knowledge by introducing a fine-grain focus signal and a focus-constrained attention mechanism to take full advantage of focus. |
| Outcome: | The proposed model can generate more diverse and informative responses compared with state-of-the-art models. |
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, document summarization is challenging even for humans. |
| Approach: | They propose a focus attention mechanism which encourages decoders to generate tokens that are topically similar to the input document. |
| Outcome: | The proposed method outperforms top-k and nucleus sampling methods on the BBC extreme summarization task and is more accurate than focus attention-based models. |
AMR Parsing as Sequence-to-Graph Transduction (P19-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsing is the task of transducing natural language text into AMR, a graphbased formalism used for capturing sentence-level semantics. |
| Approach: | They propose a model that treats AMR parsing as sequence-to-graph transduction by aligner-free, and can be effectively trained with limited amounts of labeled AMR data. |
| Outcome: | The proposed model outperforms all previously reported SMATCH scores on AMR 2.0 (76.3%) and AMR 1.0 (70.2%). |
Enriched In-Order Linearization for Faster Sequence-to-Sequence Constituent Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | Sequence-to-sequence constituent parsing requires a linearization to represent trees as sequences. Top-down tree linearizations have achieved the best accuracy to date. |
| Approach: | They propose to use an in-order shift-reduce linearization instead of a top-down tree linearization to represent trees as sequences. |
| Outcome: | The proposed approach achieves the best accuracy to date on the English PTB dataset among fully-supervised single-model sequence-to-sequence constituent parsers. |
ABC: Attention with Bounded-memory Control (2022.acl-long)
Copied to clipboard
Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, Noah A. Smith
| Challenge: | Existing approaches to attention with bounded-memory control (ABC) have a quadratic complexity in sequence lengths, making it prohibitive for long sequences. |
| Approach: | They propose a new abstraction that bounds memory size to improve efficiency . they propose bounded-memory control, which connects several efficient attention variants . |
| Outcome: | The proposed approach outperforms existing approaches on language modeling, machine translation, and masked language model finetuning. |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
Dynamic Feature Selection with Attention in Incremental Parsing (C18-1)
Copied to clipboard
| Challenge: | Currently, incremental transition-based parsers require that all inputs are visible from the beginning to extract good features from a limited local context. |
| Approach: | They propose a technique to maximize local features with an attention mechanism which works as context- dependent dynamic feature selection. |
| Outcome: | The proposed technique can extract features from a limited local context and is able to perform multilingual experiments and demon strate on local ambiguous points. |