A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing deep learning models have the attention mechanism to improve performance, but the inherent characteristics of deep learning model complexity and the flexibility of the attention structure make them difficult to explain. |
| Approach: | They propose a two-tier attention architecture to decouple the complexity of explanation and the decision-making process by using large-scale news corpora. |
| Outcome: | The proposed model can achieve competitive performance with state-of-the-art models and illustrates its appropriateness from an explainability perspective. |
Similar Papers
Effective Attention Modeling for Aspect-Level Sentiment Classification (C18-1)
Copied to clipboard
| Challenge: | Aspect-level sentiment classification aims to determine sentiment polarity of review sentence towards opinion target . main challenge is to separate different opinion contexts for different targets . |
| Approach: | They propose a method that captures the semantic meaning of the opinion target and a model that incorporates syntactic information into the attention mechanism. |
| Outcome: | The proposed method captures the semantic meaning of the opinion target and incorporates syntactic information into the attention mechanism. |
A Hierarchical Neural Attention-based Text Classifier (D18-1)
Copied to clipboard
| Challenge: | Existing hierarchical classification models are unable to handle large corpora and the number of categories increases with increasing corpus. |
| Approach: | They propose to use external knowledge to introduce a hierarchical neural attention-based classifier to help with the classification of documents. |
| Outcome: | The proposed model performs better than or comparable to state-of-the-art hierarchical models at significantly lower computational cost while maintaining high interpretability. |
Is Attention Explanation? An Introduction to the Debate (2022.acl-long)
Copied to clipboard
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, Patrick Watrin
| Challenge: | Attention has been used in various tasks of NLP and other fields of machine learning to increase performance and provide some explanations. |
| Approach: | They propose to use attention as an explanation for deep learning models to increase performance . they propose to apply attention weights to queries and queries based on scalar scores . |
| Outcome: | The proposed model can be used to increase performance while providing some explanations. |
Rethinking Self-Attention: Towards Interpretability in Neural Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that attention mechanisms provide arguably explainable attention distributions that can help to interpret predictions. |
| Approach: | They propose a new self-attention layer where attention heads represent labels. |
| Outcome: | The proposed model obtains state-of-the-art results on the Penn Treebank and Chinese Treebank. |
Attention and Lexicon Regularized LSTM for Aspect-based Sentiment Analysis (P19-2)
Copied to clipboard
| Challenge: | End-to-end deep learning systems lack flexibility as one cannot adjust the network to fix an obvious problem. |
| Approach: | They propose a way to leverage lexicon information to make the model more flexible . they also explore the effect of regularizing attention vectors to allow the network to have a broader "focus" |
| Outcome: | The proposed approach leverages lexicon information to make it more flexible and robust. |
Is Attention Interpretable? (P19-1)
Copied to clipboard
| Challenge: | Attention mechanisms have recently boosted performance on a range of NLP tasks. |
| Approach: | They propose to manipulate attention weights in text classification models and analyze the resulting differences in their predictions. |
| Outcome: | The proposed approach improves models' predictions by using gradient-based rankings of attention weights. |
Simple Attention-Based Representation Learning for Ranking Short Social Media Posts (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to ranking short social media posts are complex and require different components to capture a multitude of relevance signals. |
| Approach: | They propose a word-level Siamese architecture with attention-based mechanisms for capturing semantic "soft" matches between query and post tokens. |
| Outcome: | The proposed model is faster and simpler than existing models and more efficient than existing approaches. |
Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to label large-scale data are inadequate for distantly supervised relation extraction (DS-RE). |
| Approach: | They propose a multi-level structured (2-D matrix) self-attention mechanism for DS-RE using bidirectional recurrent neural networks. |
| Outcome: | The proposed framework significantly outperforms baselines on two publicly available DS-RE datasets in terms of PR curves, P@N and F1 measures. |
How does Attention Affect the Model? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on the effectiveness of attention in NLP do not consider changes in semantic capability of different components. |
| Approach: | They propose a framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
| Outcome: | The proposed framework exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines indicators to capture the impact of attention on sequence semantic. |
Towards Transparent and Explainable Attention Models (2020.acl-main)
Copied to clipboard
Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, Balaraman Ravindran
| Challenge: | Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model’s predictions. |
| Approach: | They propose to modify LSTM cells to ensure that the hidden representations learned at different time steps are diverse. |
| Outcome: | The proposed model can provide a faithful explanation if a higher attention weight implies a greater impact on the model’s prediction. |