Papers by Yuval Ran-Milo
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2025) show a tendency to allocate disproportionate attention mass to early (often first) positions independent of semantic content. |
| Approach: | They find that Transformers display an attention sink: disproportionate attention to the first position. |
| Outcome: | The proposed sinks are found in GPT-2–style models with learned query biases and absolute positional embeddings. |
Mamba Knockout for Unraveling Factual Information Flow (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has introduced Mamba-based SSM architectures that rival Transformer performance in various settings. |
| Approach: | They propose to use attentional interpretability techniques originally developed for Transformers to trace how information is transmitted and localized across tokens and layers. |
| Outcome: | The proposed model disentangles how distinct features enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations. |
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2024) show that softmax models display an attention sink . he argues that normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state. |
| Approach: | They show that normalization over a trigger-conditional behavior *necessarily* induces a sink in softmax self-attention models. |
| Outcome: | The proposed model can solve a task with no sink in softmax models. |