DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers

Challenge:	Existing interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity.
Approach:	They propose a method to analyze encoder-decoder Transformers by using the decoder module Model Output encoder to cross-attend representations of intermediate encoder activations instead of using the default output.
Outcome:	The proposed method maps uninterpretable representations to human-interpreted sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models.

Similar Papers

Overview

Challenge

Approach

Outcome

Transformer-specific Interpretability (2024.eacl-tutorials)

Copied to clipboard

Hosein Mohebbi, Jaap Jumelet, Michael Hanna, Afra Alishahi, Willem Zuidema

Explainability & Interpretability in NLP

Challenge:	Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque.
Approach:	This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions .
Outcome:	This tutorial aims to show how a new trending approach can be applied to Transformer-based models.

On the Sub-layer Functionalities of Transformer Decoder (2020.findings-emnlp)

Copied to clipboard

Yilin Yang, Longyue Wang, Shuming Shi, Prasad Tadepalli, Stefan Lee, Zhaopeng Tu

Language Models

Challenge:	Existing efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation have focused on assessing the encoded representations or interpreting the multi-head self-attentions.
Approach:	They propose to use Transformer-based encoder-decoder architectures to analyze how information is propagated through each module of each decoder layer.
Outcome:	The proposed model can be dropped with minimal loss of performance on three translation datasets and can be used to train and inference faster.

DecompX: Explaining Transformers Decisions by Propagating Token Decomposition (2023.acl-long)

Copied to clipboard

Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar

Explainability & Interpretability in NLP

Challenge:	Existing vector-based explanation methods for Transformer-based models are limited in their ability to explain the decisions of multiple layers.
Approach:	They propose a vector-based explanation method based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers.
Outcome:	The proposed method outperforms existing vector-based and gradient-based methods on transformer-based models by a wide margin.

InterpreT: An Interactive Visualization Tool for Interpreting Transformers (2021.eacl-demos)

Copied to clipboard

Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, Moshe Wasserblat

Semantic Text Processing

Challenge:	Using Transformer-based models for NLU/NLP tasks is a growing interest . but there are many open questions regarding the behavior of these models .
Approach:	They present an interactive visualization tool for interpreting Transformer-based models.
Outcome:	The tool can track and visualize token embeddings through each layer of a Transformer, highlight distances between certain token embeds, and identify task-related functions of attention heads using new metrics.

Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer (2022.emnlp-main)

Copied to clipboard

Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, Marta R. Costa-jussà

Language Models

Challenge:	Neural Machine Translation (NMT) relies on source sentence and target prefix attributions for each input token.
Approach:	They propose an interpretability method that tracks input tokens’ attributions for both contexts and extends it to any encoder-decoder Transformer-based model.
Outcome:	The proposed method can be extended to any encoder-decoder Transformer-based model and provides insights into their behaviour.

Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)

Copied to clipboard

Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort

Semantic Text Processing

Challenge:	Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond.
Approach:	They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss.
Outcome:	The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy.

ReadOnce Transformers: Reusable Representations of Text for Transformers (2021.acl-long)

Copied to clipboard

Shih-Ting Lin, Ashish Sabharwal, Tushar Khot

Semantic Text Processing

Challenge:	ReadOnce Transformers is a task-independent, task-dependent, and compressed representation of text.
Approach:	They propose a transformer-based model that can build an information-capturing, task-independent, and compressed representation of text.
Outcome:	The proposed model can build an information-capturing, task-independent, and compressed representation of text.

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)

Copied to clipboard

Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, Chris Dyer

Language Models

Challenge:	a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions.
Approach:	They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions.
Outcome:	The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics.

Interpretability-based Tailored Knowledge Editing in Transformers (2024.emnlp-main)

Copied to clipboard

Yihuai Hong, Aldo Lipani

Responsible & Trustworthy NLP

Challenge:	Existing methods for modifying in-context learning fail to analyze the instability of in-constitu learning outcomes.
Approach:	They propose a model-based knowledge editing method that considers the unique information flow of each sample and aims to correct errors without costly retraining.
Outcome:	The proposed method exploits the critical role of feed-forward MLPs in decoder-only models and reveals diverse attribute recall across transformer layers, guiding edits to specific features at different depths and mitigating over-editing issues.

Inseq: An Interpretability Toolkit for Sequence Generation Models (2023.acl-demo)

Copied to clipboard

Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar van der Wal

Semantic Text Processing

Challenge:	Recent studies focused on classification tasks while largely overlooking generation settings due to a lack of dedicated tools.
Approach:	They propose to use Inseq to democratize access to interpretability analyses of sequence generation models by enabling intuitive extraction of models’ internal information and feature importance scores for popular decoder-only and encoder-decoder Transformers architectures.
Outcome:	The proposed library can extract models’ internal information and feature importance scores for popular decoder-only and encoder-decoder Transformers architectures.