Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, Balaraman Ravindran
| Challenge: | Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model’s predictions. |
| Approach: | They propose to modify LSTM cells to ensure that the hidden representations learned at different time steps are diverse. |
| Outcome: | The proposed model can provide a faithful explanation if a higher attention weight implies a greater impact on the model’s prediction. |
Similar Papers
Attention is not Explanation (N19-1)
Copied to clipboard
| Challenge: | Attention mechanisms have seen wide adoption in neural NLP models. |
| Approach: | They perform extensive experiments to assess the degree to which attention weights provide meaningful "explanations" they find that attention weighted inputs are often uncorrelated with gradient-based measures of feature importance . |
| Outcome: | The proposed model is based on a distribution over attended-to input units . the findings show that attention weights are often uncorrelated with features . |
Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that attention cannot be considered as a faithful explanation across encoders and tasks. |
| Approach: | They propose a new family of Task-Scaling mechanisms that scale attention weights across tasks and two attention mechanisms. |
| Outcome: | The proposed models improve explanation faithfulness across two attention mechanisms, five encoders and five text classification datasets without sacrificing predictive performance. |
Is Attention Interpretable? (P19-1)
Copied to clipboard
| Challenge: | Attention mechanisms have recently boosted performance on a range of NLP tasks. |
| Approach: | They propose to manipulate attention weights in text classification models and analyze the resulting differences in their predictions. |
| Outcome: | The proposed approach improves models' predictions by using gradient-based rankings of attention weights. |
Learning to Deceive with Attention-Based Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | Attention mechanisms are ubiquitous components in neural network architectures and are often claimed to confer interpretability. |
| Approach: | They propose a method for training models to produce deceptive attention masks by combining weights assigned to designated impermissible tokens with a weighted sum. |
| Outcome: | The proposed method reduces the weight assigned to designated impermissible tokens while still using them across multiple models and tasks. |
Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | In this paper, we examine the behavior of deep learning models in their intermediate layers . saliency determines what is critical for the final decision of a deep model . |
| Approach: | They propose to interpret the intermediate layers of deep models by visualizing the saliency of attention and LSTM gating signals. |
| Outcome: | The proposed methods reveal interesting insights and identify critical information contributing to the model decisions. |
Attention is not not Explanation (D19-1)
Copied to clipboard
| Challenge: | Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. |
| Approach: | They propose to use a simple uniform-weights baseline, a variance calibration and a diagnostic framework to determine when/whether attention can be used as explanation in RNN models. |
| Outcome: | The proposed tests show that even reliable adversarial distributions don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability. |
Exploring Attention Attractors in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have suggested that attention attractors function as "summary tokens" while others speculate that tokens with weaker semantics attract high attention, they act as attention sinks that offload excessive attention. |
| Approach: | They examine attention attractors, tokens that draw significantly high attention, in large language models. |
| Outcome: | The proposed models are able to capture long-range dependencies within a given context. |
Is Sparse Attention more Interpretable? (2021.acl-short)
Copied to clipboard
| Challenge: | Sparse attention has been claimed to increase model interpretability . however, the attention distribution is typically over representations internal to the model rather than the inputs themselves . |
| Approach: | They conduct experiments to understand how sparsity affects our ability to use attention as an explainability tool. |
| Outcome: | The proposed model does not map to a sparse set of influential inputs, but rather to fewer inputs. |
Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining (2022.findings-emnlp)
Copied to clipboard
| Challenge: | To explain NLP models, importance measures are often used to inform input tokens are important for making a prediction. |
| Approach: | They propose a faithfulness metric that masks allegedly important tokens and retrains the model. |
| Outcome: | The proposed metric is based on LSTM-attention models and RoBERTa models. |
Attention and Lexicon Regularized LSTM for Aspect-based Sentiment Analysis (P19-2)
Copied to clipboard
| Challenge: | End-to-end deep learning systems lack flexibility as one cannot adjust the network to fix an obvious problem. |
| Approach: | They propose a way to leverage lexicon information to make the model more flexible . they also explore the effect of regularizing attention vectors to allow the network to have a broader "focus" |
| Outcome: | The proposed approach leverages lexicon information to make it more flexible and robust. |