| Challenge: | In computer vision, the trigger can be a fixed pattern overlaid on the images or videos. |
| Approach: | They propose an attention-based Trojan detector to distinguish Trojaned models from clean ones by observing the attention focus drifting behavior of Trojanes. |
| Outcome: | The proposed detector is based on transformer’s attention and can distinguish Trojan models from clean ones. |
Similar Papers
Attention-Enhancing Backdoor Attacks Against BERT-based Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing textual backdoor attacks focus on generating stealthy triggers or modifying model weights. |
| Approach: | They propose a Trojan Attention Loss (TAL) which enhances the Trojan behavior by directly manipulating attention patterns. |
| Outcome: | The proposed method improves the effectiveness of the backdoor attacks on different backbone models and tasks. |
How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work on multi-head attention mechanism shows heuristics and clues in analyzing various aspects of the mechanism. |
| Approach: | They propose to cluster attention heatmaps into significantly different patterns through unsupervised clustering on top of a set of proposed features. |
| Outcome: | The proposed features can explain and calibrate different attention heads in Transformer models. |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work discouraging the use of attention distributions for explaining a model’s behaviour suggests that attention distribution can provide insights into local behaviour of attention heads. |
| Approach: | They propose a distinction between local patterns revealed by attention and global patterns that refer back to the input and analyze BERT from both angles. |
| Outcome: | The proposed model can explain local behaviour of attention heads by comparing local and global patterns from both angles. |
FLASH: Focused Layer Attention Sink Hijacking (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) remain vulnerable to jailbreaking attacks despite advances in safety alignment . |
| Approach: | They propose a new diagnostic auditing framework that dismantles the model's internal safety anchor by precisely scaling attention scores in these vulnerable layers. |
| Outcome: | The proposed framework achieves a state-of-the-art Attack Success Rate of over 77% with an unprecedented efficiency of 1.53 queries on average. |
Attention Tracker: Detecting Prompt Injection Attacks in LLMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated actions. |
| Approach: | They propose a training-free method that tracks attention patterns on instruction to detect prompt injection attacks without additional inference. |
| Outcome: | The proposed method shows an improvement of up to 10.0% over existing methods and performs well even on small LLMs. |
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based models are stretched to enormous sizes, requiring increasingly larger training datasets and unsustainable amount of compute resources. |
| Approach: | They propose an alternative compatibility function for the Transformer-based attention mechanism that exploits an overlap in the learned representation of the traditional scaled dot-product attention mechanism. |
| Outcome: | The proposed model achieves 79.36 on the GLUE benchmark against 78.74 for the traditional implementation and reduces the number of trainable parameters by 6%. |
Input-specific Attention Subnetworks for Adversarial Detection (2022.findings-acl)
Copied to clipboard
| Challenge: | a new method to prune attention heads is proposed for adversarial detection . attention heads in models such as BERT are over-provisioned and can be pruned . |
| Approach: | They propose a method to construct input-specific attention subnetworks from which three features are extracted to discriminate between authentic and adversarial inputs. |
| Outcome: | The proposed method significantly improves state-of-the-art adversarial detection accuracy on 10 NLU datasets with 11 different adversarials. |
Attention Can Reflect Syntactic Structure (If You Let It) (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has attempted to decode linguistic structure from the Transformer . but, much of the work focused on English, a language with rigid word order and a lack of inflectional morphology. |
| Approach: | They propose to fine-tune a feature encoder for BERT to learn linguistic structure from its multi-head attention mechanism. |
| Outcome: | The proposed model can decode full trees above baseline accuracy from single attention heads across languages. |