Papers with Attention
Fast Exact Retrieval for Nearest-neighbor Lookup (FERN) (2024.naacl-srw)
Copied to clipboard
| Challenge: | Exact nearest neighbor search is computationally intensive and complex . Attention has shifted towards Approximate Nearest-Neighbor (ANN) retrieval techniques . |
| Approach: | They propose an algorithm for logarithmic Fast Exact Retrieval for Nearest-neighbor lookup which achieves O(dlog N) look-up with 100% recall on 10 million d=128 uniformly generated vectors. |
| Outcome: | The proposed algorithm achieves O(dlog N) look-up with 100% recall on 10 million d=128 uniformly generated vectors. |
Dilated LSTM with attention for Classification of Suicide Notes (D19-62)
Copied to clipboard
| Challenge: | Using a dilated LSTM with attention we achieve an accuracy of 87.34% compared to baselines of 80.35% and 82.27%. |
| Approach: | They propose a dilated LSTM with attention mechanism for document-level classification of suicide notes, last statements and depressed notes. |
| Outcome: | The proposed model achieves an accuracy of 87.34% compared to baselines of 80.35% and 82.27%. |
Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that attention cannot be considered as a faithful explanation across encoders and tasks. |
| Approach: | They propose a new family of Task-Scaling mechanisms that scale attention weights across tasks and two attention mechanisms. |
| Outcome: | The proposed models improve explanation faithfulness across two attention mechanisms, five encoders and five text classification datasets without sacrificing predictive performance. |
Attention Optimization for Abstractive Document Summarization (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization models require attention to reproduce the most salient information. |
| Approach: | They propose to use local and global variances to augment the vanilla attention model to reproduce the most salient information and avoid repetitions. |
| Outcome: | The proposed attention refinement unit can reproduce the most salient information and avoid repetitions on CNN/Daily Mail dataset. |
Understanding Attention for Text Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on whether local attention weights reflect the importance of input representations. |
| Approach: | They propose to analyze for each word token the following two quantities: its polarity score and its attention score, where the latter is a global assessment on the token’s significance. |
| Outcome: | The proposed model can be improved under conditions where the interplay between the two quantities can contribute towards model performance. |
LittleBird: Efficient Faster & Longer Transformer for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for BERT have a limitation dealing with long inputs due to its attention mechanism. |
| Approach: | They propose a model based on BigBird with improved speed and memory footprint . they propose 'pack and unpack attention' to replace global attention . |
| Outcome: | The proposed model can work on long inputs even after being pre-trained on short inputs. |
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |
A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) have advanced beyond traditional Large Language Models, yet they pose heightened safety risks. |
| Approach: | They propose a first jailbreak attack targeting Large Reasoning Models . they exploit a Chaos Machine component to transform attack prompts with diverse one-to-one mappings based on the reasoning chain . |
| Outcome: | The proposed attack exploits the unique vulnerabilities of LRMs by integrating a Chaos Machine. success rates of the mousetrap attack are as high as 96%, 86% and 98% respectively. |
Attention and Edge-Label Guided Graph Convolutional Networks for Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is the recognition of entities with specific meanings in the text, mainly including person, organization, location, etc. |
| Approach: | They propose an edge-aware node joint update module and introduce a node-awful edge update module to explore hidden in structured information and solve the wrong dependency label information to some extent. |
| Outcome: | The proposed model can exploit the structured information on the dependency tree to improve the recognition of long entities. |
The Learnability of Model-Theoretic Interpretation Functions in Artificial Neural Networks (2026.findings-acl)
Copied to clipboard
| Challenge: | Entity vectors improve scores on basic event, while gated architectures benefit most. |
| Approach: | They extend entity-level semantic representations, modern architectures, principled competing event generation, extended systematicity tests and a two-dimensional difficulty analysis disaggregating results by modifier complexity. |
| Outcome: | The proposed model-theoretic interpretation functions generalize systematically to out-of-training-sample sentences. |
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020.emnlp-main)
Copied to clipboard
| Challenge: | Attention is a key component of Transformers, which have achieved considerable success in natural language processing. |
| Approach: | They propose to integrate attention weights and the norm of transformed input vectors into a norm-based analysis that incorporates the norm. |
| Outcome: | The proposed analysis shows that attention weights alone determine the output of attention and that reasonable word alignment can be extracted from attention mechanisms of Transformers. |