Papers with fMRI
Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent work has observed an inverse scaling relationship between Transformers’ per-word estimated probability and the predictive power of their surprisal estimates on reading times. |
| Approach: | They conducted a more comprehensive evaluation using surprisal estimates from 17 pre-trained LMs on two functional magnetic resonance imaging datasets. |
| Outcome: | Recent work shows that surprisal from larger Transformer-based models is less predictive of reading times, resolving the inconclusive results and indicating that this trend is not specific to latency-based measures. |
Using the RUPEX Multichannel Corpus in a Pilot fMRI Study on Speech Disfluencies (2020.lrec-1)
Copied to clipboard
Katerina Smirnova, Nikolay Korotaev, Yana Panikratova, Irina Lebedeva, Ekaterina Pechenkova, Olga Fedorova
| Challenge: | Numerous classifications of disfluencies have been proposed and/or implemented in annotating speech corpora. |
| Approach: | They propose to use Russian multichannel corpus RUPEX to create fragments of speech disfluencies and their clusters. |
| Outcome: | The proposed method allows to create fragments in terms of requirements for the fMRI BOLD temporal resolution. |
Linking artificial and human neural representations of language (D19-1)
Copied to clipboard
| Challenge: | a pre-trained BERT architecture is used to fine-tune sentence encoding models on a variety of natural language understanding (NLU) tasks. |
| Approach: | They compare sentence encoding models with fMRI-based fMR predictions of the sentence . they use a pre-trained BERT architecture as a baseline and fine-tune it on a variety of natural language understanding (NLU) tasks. |
| Outcome: | The proposed model does not yield significant improvements in brain decoding performance on the natural language understanding (NLU) tasks. |
Multimodal Corpus of Bidirectional Conversation of Human-human and Human-robot Interaction during fMRI Scanning (2020.lrec-1)
Copied to clipboard
Birgit Rauchbauer, Youssef Hmamouche, Brigitte Bigi, Laurent Prévot, Magalie Ochs, Thierry Chaminade
| Challenge: | a study of real-life bi-directional conversations combines multimodal corpus with neural, physiological and behavioral data. |
| Approach: | They propose a multimodal corpus derived from natural conversations . they used human-human interactions as a control condition . |
| Outcome: | The proposed corpus includes neural, physiological and behavioral data. |
BrainPredict: a Tool for Predicting and Visualising Local Brain Activity (2020.lrec-1)
Copied to clipboard
| Challenge: | Using fMRI, we recorded a corpus of human-human and human-robot conversations while participants brain activity was recorded with f.MRI, but we did not find any tools for displaying together brain activity prediction of non-controlled conversations, the raw material used in this prediction and the features used for these predictions. |
| Approach: | They propose a tool that allows dynamic prediction and visualization of an individual’s local brain activity during a conversation using raw behavioral data. |
| Outcome: | The proposed tool takes as input behavioral features computed from raw data, mainly the participant and the interlocutor speech but also the participant’s visual input and eye movements. |
Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate that this method effectively disentangles the information within fMRI signals. |
| Approach: | They propose a task Memory Disentangling which extracts and decodes past information from fMRI signals. |
| Outcome: | The proposed method extracts and decodes past information from fMRI signals. |
Vision-Language Models Align with Human Neural Representations in Concept Processing (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that transformer-based vision-language models capture the multimodality of concept processing in the human brain. |
| Approach: | They analysed multiple VLMs employing different strategies to integrate visual and textual modalities, along with language-only counterparts. |
| Outcome: | The transformer-based vision-language models outperform language-only models in two experimental conditions, while only some outperformed the language-based models. |
Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations (C18-1)
Copied to clipboard
| Challenge: | Neural activation models have been proposed to map word semantics to localized neural activations. |
| Approach: | They propose a computational model that estimates semantic similarity in the neural activation space and investigate its performance for various natural language processing tasks. |
| Outcome: | The proposed model performs better than state-of-the-art word embeddings for the task of semantic similarity estimation between very similar or very dissimilar words while performing well on other tasks such as entailment and word categorization. |
CDRNN: Discovering Complex Dynamics in Human Language Processing (2021.acl-long)
Copied to clipboard
| Challenge: | Behavioral and fMRI experiments reveal detailed and plausible estimates of human language processing dynamics . central questions in psycholinguistics concern the mental processes involved in incremental human language understanding . |
| Approach: | They propose a continuous-time deconvolutional regressive neural network that captures time-varying, non-linear, and delayed influences of predictors on the response. |
| Outcome: | The proposed neural network captures time-varying, non-linear, and delayed influences on the response . Behavioral and fMRI experiments show it generalizes better than baselines . |
Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have successfully decoded natural language from non-invasive brain signals . current dataset splitting methods suffer from data leakage problem . |
| Approach: | They propose a right cross-subject data splitting criterion without data leakage for decoding fMRI and EEG signal to text. |
| Outcome: | The proposed method overfits and overestimates brain-to-text decoding models. |
Surprisal Estimators for Human Reading Times Need Character Models (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that character models can be applied to a structural parser-based processing model to calculate word generation probabilities. |
| Approach: | They propose to use a character model to calculate word generation probabilities from a structural parser-based processing model. |
| Outcome: | The proposed model performs better on self-paced reading, eye-tracking, and fMRI data than large-scale language models trained on much more data. |
Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a popular approach to decompose the neural bases of language requires large and costly data sets to obtain. |
| Approach: | They propose a model-based approach to decompose the neural bases of language that can be used to correlate brain responses to different stimuli. |
| Outcome: | The proposed model-based approach replicates the seminal study of Lerner et al. (2011), which revealed the hierarchy of language areas by comparing the functional-magnetic resonance imaging (fMRI) of seven subjects listening to 7min of both regular and scrambled narratives. |
Attention weights accurately predict language representations in the brain (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In Transformer-based language models, the attention mechanism converts token embeddings into contextual embeddables that incorporate information from neighboring words. |
| Approach: | They analyze fMRI recordings of English language learners and extract attention weights from them to determine how well they can predict brain responses. |
| Outcome: | The resulting hidden state embeddings are more accurate than lexical embeddngs or RNN-based models. |
Speech language models lack important brain-relevant semantics (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that text-based language models predict both text- and speech-evoked brain activity. |
| Approach: | They remove low-level stimulus features from language models to assess their impact on alignment with fMRI brain recordings during reading and listening. |
| Outcome: | The proposed model removes low-level features from fMRI brain recordings to assess their impact on alignment with fmr recordings. |
Modeling Affirmative and Negated Action Processing in the Brain with Lexical and Compositional Semantic Models (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that distributional semantic models can be used to decode fMRI patterns associated with specific aspects of semantic composition, such as the negation function. |
| Approach: | They apply lexical and compositional semantic models to decode fMRI patterns associated with negated and affirmative sentences containing hand-action verbs. |
| Outcome: | The proposed models show reduced decoding of sentences where the verb is in the negated context, as compared to the affirmative one, within brain regions implicated in action-semantic processing. |
Is the Brain Mechanism for Hierarchical Structure Building Universal Across Languages? An fMRI Study of Chinese and English (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that the brain builds hierarchical syntactic structures, but it is unknown whether they are universal across languages. |
| Approach: | They analyze the working memory requirements when applying parsing strategies to two languages: Chinese and English. |
| Outcome: | The proposed method shows that the brain adopts parsing strategies with less memory load according to different language structures. |
Higher-order Comparisons of Sentence Encoder Representations (D19-1)
Copied to clipboard
| Challenge: | a technique developed by neuroscientists compares activity patterns of different measurement modalities . a recent study examined the correspondence between popular pretrained language encoders and human processing difficulty . |
| Approach: | They employ a technique to compare activity patterns of different measurement modalities . they establish a correspondence between widely-employed pretrained language encoders and human processing difficulty . |
| Outcome: | The proposed technique can be used to compare representational geometries of neural models . it does not require large training samples and is not prone to overfitting, authors say . |
UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on decoding word-level fMRI volumes from a restricted vocabulary. |
| Approach: | They propose an open-vocabulary task to bridge fMRI time series and human language . they use a pre-trained language model to construct a robust encoder for cognitive signals . |
| Outcome: | The proposed task bridges fMRI time series and human language with a baseline model. |
Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models (2024.emnlp-main)
Copied to clipboard
Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Yamaguchi, Rieko Kubo, Shinji Nishimoto, Yu Takagi
| Challenge: | Recent studies have assessed different levels of semantic content, such as speech, objects, and stories, separately. |
| Approach: | They used functional magnetic resonance imaging to record brain activity while watching 8.3 hours of dramas and movies. |
| Outcome: | The findings show that LLMs predict human brain activity more accurately than traditional language models, particularly for complex background stories. |
BrainLoc: Brain Signal-Based Object Detection with Multi-modal Alignment (2025.findings-emnlp)
Copied to clipboard
Jiaqi Duan, Xiaoda Yang, Kaixuan Luan, Hongshun Qiu, Weicai Yan, Xueyi Zhang, Youliang Zhang, Zhaoyang Li, Donglin Huang, JunYu Lu, Ziyue Jiang, Xifeng Yang
| Challenge: | BrainLoc is a lightweight object detection model guided by fMRI signals. |
| Approach: | They propose a brain-based object detection model guided by fMRI signals . they employ a multi-modal alignment strategy that enhances fmr feature extraction . |
| Outcome: | The proposed model improves fMRI-based object detection accuracy and convenience. |
Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals (2026.acl-long)
Copied to clipboard
| Challenge: | Language models have emerged as powerful tools for predicting human brain activity during language comprehension. |
| Approach: | They propose a technique that leverages electrocorticography’s millisecond precision to train speech language models. |
| Outcome: | The proposed technique improves brain alignment over pretrained and distillation models and produces higher gains in higher-order language regions. |