Papers by Mohsen Fayyaz
DecompX: Explaining Transformers Decisions by Propagating Token Decomposition (2023.acl-long)
Copied to clipboard
| Challenge: | Existing vector-based explanation methods for Transformer-based models are limited in their ability to explain the decisions of multiple layers. |
| Approach: | They propose a vector-based explanation method based on the construction of decomposed token representations and their successive propagation throughout the model without mixing them in between layers. |
| Outcome: | The proposed method outperforms existing vector-based and gradient-based methods on transformer-based models by a wide margin. |
Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence (2025.acl-long)
Copied to clipboard
| Challenge: | Notably, when multiple biases combine, models exhibit catastrophic performance degradation, selecting the answer-containing document in less than 10% of cases over a synthetic biased document without the answer. |
| Approach: | They repurpose a relation extraction dataset to quantify the impact of heuristic biases on retrievers like Dragon+ and Contriever. |
| Outcome: | The proposed models exhibit catastrophic performance degradation when multiple biases combine, selecting the answer-containing document in less than 10% of cases over a synthetic biased document without the answer. |
GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for interpreting the underlying dynamics of Transformers have been criticized for their lack of reliability. |
| Approach: | They propose a token attribution analysis method that incorporates all components in the encoder block and aggregates this across layers. |
| Outcome: | The proposed method significantly outperforms existing methods on saliency scores and correlation with gradient-based salience scores. |
Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models assume they encode metaphorical knowledge useful for NLP systems. |
| Approach: | They propose to probing metaphoricity information in PLMs and measure their generalization . they find that contextual representations in PMLs encode metaphorical knowledge . |
| Outcome: | The proposed model can encode metaphorical knowledge across languages and datasets . the model can be used to train and test NLP systems . |