Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that attention heads have a temporal induction property that allows them to learn and reproduce sequences of tokens. |
| Approach: | They analyze attention heads and transformer outputs to examine in-context temporal biases . they find that transformer output has a tendency toward in-constext serial recall . |
| Outcome: | The findings shed light on similarities and differences between LLMs and human memory and learning. |
Similar Papers
Beyond Semantics: How Temporal Biases Shapes Retrieval in Transformer and State-Space Models (2026.eacl-long)
Copied to clipboard
| Challenge: | In-context learning depends on what appears in the prompt and on when it appears. |
| Approach: | They construct prompts with repeated anchor tokens and average their predictions over hundreds of random permutations of the intervening context. |
| Outcome: | The proposed model predicts the same tokens over hundreds of permutations over time. |
Beyond Position: the emergence of wavelet-like properties in Transformers (2025.acl-long)
Copied to clipboard
| Challenge: | Despite its widespread adoption, theoretical limitations in positional encodings are resolved by developing emergent, wavelet-like processing strategies. |
| Approach: | They propose to use Rotary Position Embeddings to develop emergent, wavelet-like properties that compensate for the positional encoding’s theoretical limitations. |
| Outcome: | The attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Telling BERT’s Full Story: from Local Attention to Global Aggregation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work discouraging the use of attention distributions for explaining a model’s behaviour suggests that attention distribution can provide insights into local behaviour of attention heads. |
| Approach: | They propose a distinction between local patterns revealed by attention and global patterns that refer back to the input and analyze BERT from both angles. |
| Outcome: | The proposed model can explain local behaviour of attention heads by comparing local and global patterns from both angles. |
Linear Recency Bias During Training Improves Transformers’ Fit to Reading Times (2025.coling-main)
Copied to clipboard
| Challenge: | Recent research has shown a strong fit between surprisal values from Transformers and reading times. |
| Approach: | They evaluate a Transformer model that uses a recency bias added to attention scores to improve the fit to human reading times. |
| Outcome: | The proposed model improves on a Transformer that includes a recency bias added to attention scores. |
Identifying Semantic Induction Heads to Understand In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance, but lack of transparency in their inference logic raises concerns about their trustworthiness. |
| Approach: | They conduct a detailed analysis of the operations of attention heads to understand their in-context learning of LLMs. |
| Outcome: | The proposed analysis of attention heads reveals that they increase the output logits of object tokens and recall objects . the proposed model is a novel approach to understand the in-context learning of large language models. |
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm . a novel decomposition of Transformer output embeddables is demonstrated . |
| Approach: | They propose to decompose Transformer output embeddings into a sum of vector factors . they show multi-head attentions and feed-forwards are not equally useful in downstream applications . |
| Outcome: | The proposed method outperforms recurrent architectures on a wide variety of tasks. |
Characterizing the Expressivity of Local Attention in Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that global and local attention are expressively complementary. |
| Approach: | They propose to restrict global attention to a fixed-size window of preceding tokens . they also propose to add local attention to local-only transformers to increase model quality . |
| Outcome: | The proposed model outperforms the global–local transformers on formal language recognition and natural language modeling. |
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has identified MLP modules in early layers as key contributors to factual recall. |
| Approach: | They conduct a comprehensive evaluation of factual recall across several models . they find that Qwen-based models behave differently from previous patterns . |
| Outcome: | The results show that attention modules in the earliest layers contribute more to factual recall than MLP modules. |