Linear Recency Bias During Training Improves Transformers’ Fit to Reading Times (2025.coling-main)
Copied to clipboard
| Challenge: | Recent research has shown a strong fit between surprisal values from Transformers and reading times. |
| Approach: | They evaluate a Transformer model that uses a recency bias added to attention scores to improve the fit to human reading times. |
| Outcome: | The proposed model improves on a Transformer that includes a recency bias added to attention scores. |
Similar Papers
Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies have shown that larger pre-trained language models with more parameters and lower perplexity are less predictive of human reading times. |
| Approach: | They propose to use a transformer-based model with more parameters and lower perplexity to investigate why these models are less predictive of human reading times. |
| Outcome: | The results show that the larger models with more parameters and lower perplexity are less predictive of human reading times and eye-gaze durations collected during naturalistic reading. |
Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times. |
| Approach: | They propose to evaluate surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. |
| Outcome: | The proposed model variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, while smaller models show a ‘tipping point’ at convergence after the decrease in language model perplexity . |
Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent work has observed an inverse scaling relationship between Transformers’ per-word estimated probability and the predictive power of their surprisal estimates on reading times. |
| Approach: | They conducted a more comprehensive evaluation using surprisal estimates from 17 pre-trained LMs on two functional magnetic resonance imaging datasets. |
| Outcome: | Recent work shows that surprisal from larger Transformer-based models is less predictive of reading times, resolving the inconclusive results and indicating that this trend is not specific to latency-based measures. |
Frequency Explains the Inverse Correlation of Large Language Models’ Size, Training Data Amount, and Surprisal’s Fit to Reading Times (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. |
| Approach: | They present a series of analyses showing that word frequency is a key explanatory factor underlying these two trends. |
| Outcome: | The results show that word frequency is a key explanatory factor underlying these two trends. |
Temperature-scaling surprisal estimates improve fit to human reading times – but does it do so for the “right reasons”? (2024.acl-long)
Copied to clipboard
| Challenge: | a wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word’s negative log probability in context. |
| Approach: | They propose to use large language models to predict the surprisal of a word's negative log probability in context to test their predictive power. |
| Outcome: | The proposed model can be significantly more accurate than humans because it has more data. |
The Linearity of the Effect of Surprisal on Reading Times across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large amount of insight into human language processing can be gleaned by studying word-by-word processing difficulty. |
| Approach: | They extend the study by examining eyetracking corpora of seven languages . they find evidence for superlinearity in some languages, but highly sensitive to language models . |
| Outcome: | The study extends existing studies on english to Danish, Dutch, English, German, Japanese, Mandarin, and Russian. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that attention heads have a temporal induction property that allows them to learn and reproduce sequences of tokens. |
| Approach: | They analyze attention heads and transformer outputs to examine in-context temporal biases . they find that transformer output has a tendency toward in-constext serial recall . |
| Outcome: | The findings shed light on similarities and differences between LLMs and human memory and learning. |
Reverse-Engineering the Reader (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have sought to determine to what extent language models can serve as useful models of human cognition by aligning them to human psychometric data. |
| Approach: | They propose a method to fine-tune a language model to implicitly optimize parameters of a linear regressor that directly predicts humans’ reading times of in-context linguistic units. |
| Outcome: | The proposed technique improves language models’ psychometric predictive power but also its perplexity on held-out test data. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |