BrainPredict: a Tool for Predicting and Visualising Local Brain Activity (2020.lrec-1)
Copied to clipboard
| Challenge: | Using fMRI, we recorded a corpus of human-human and human-robot conversations while participants brain activity was recorded with f.MRI, but we did not find any tools for displaying together brain activity prediction of non-controlled conversations, the raw material used in this prediction and the features used for these predictions. |
| Approach: | They propose a tool that allows dynamic prediction and visualization of an individual’s local brain activity during a conversation using raw behavioral data. |
| Outcome: | The proposed tool takes as input behavioral features computed from raw data, mainly the participant and the interlocutor speech but also the participant’s visual input and eye movements. |
Similar Papers
The Brain-IHM Dataset: a New Resource for Studying the Brain Basis of Human-Human and Human-Machine Conversations (2020.lrec-1)
Copied to clipboard
| Challenge: | Using a dataset of controlled interactions, we have studied the feedback items produced by the interlocutors during a conversation. |
| Approach: | They propose to use a dataset of controlled interactions to study feedback items and a virtual reality context to re-synthesize the conversations. |
| Outcome: | The proposed dataset compares human-human and human-machine production of feedbacks and is the first of its kind. |
Encoding and Decoding Language in the Brain with Language Models (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and brain-based fine-caching with language models. |
| Approach: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and scaling with language models. |
| Outcome: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and decoding with language models. |
Computational Linguistics for Brain Encoding and Decoding: Principles, Practices and Beyond (2024.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will explore the potential of computational linguistics to help understand brain language processing. |
| Approach: | This tutorial will explore the principles and practices of using computational linguistics methods for brain encoding and decoding. |
| Outcome: | This tutorial will explore the principles and practices of using computational linguistics methods for brain encoding and decoding. |
SpanPredict: Extraction of Predictive Document Spans with Neural Attention (2021.naacl-main)
Copied to clipboard
| Challenge: | identifying predictive text in clinical notes can be as important as the predictions themselves . identifying specific content in clinical note descriptions may illuminate previously unknown risk factors . |
| Approach: | They propose a method for identifying predictive text in clinical notes . they use linear attention to formalize the problem as predictive extraction . |
| Outcome: | The proposed model preserves differentiability and allows scalable inference via stochastic gradient descent. |
A Neural, Interactive-predictive System for Multimodal Sequence to Sequence Tasks (P19-3)
Copied to clipboard
| Challenge: | a neural interactive-predictive system is used to tackle multimodal sequence to sequence tasks . it generates text predictions to different sequence to sequencing tasks, including machine translation, image and video captioning. |
| Approach: | They present a neural interactive-predictive system for tackling multimodal sequence to sequence tasks. |
| Outcome: | The proposed system reduces human effort during the correction process by providing alternative hypotheses. |
Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals (2025.acl-long)
Copied to clipboard
| Challenge: | Existing systems for human-machine conversations are limited in predicting turn-taking and backchannel actions. |
| Approach: | They propose a multi-modal face-to-face (MM-F2F) human conversation dataset . they collect and annotate over 210 hours of human conversation videos . |
| Outcome: | The proposed model achieves state-of-the-art on turn-taking and backchannel prediction tasks. |
Multi-view and Cross-view Brain Decoding (2022.coling-1)
Copied to clipboard
| Challenge: | a recent study has shown that brain decoding models can decode concepts from single view . a multi-view decoder can take brain recordings for any view as input and predict the concept . |
| Approach: | They propose to build a multi-view decoder that can take brain recordings for any view as input and predict the concept. |
| Outcome: | The proposed systems can decode concepts from brain recordings from any view . the proposed systems have 0.68 pairwise accuracy across view pairs and 0.8 average pairwise precision across tasks. |
InteractSpeech: A Speech Dialogue Interaction Corpus for Spoken Dialogue Model (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Spoken Dialogue models face challenges in handling nuanced interactional phenomena, such as interruptions and backchannels. |
| Approach: | They propose to use a 150-hour English speech interaction dialogue dataset to empower spoken dialogue models with nuanced real-time interaction capabilities. |
| Outcome: | The proposed dataset trains and evaluates a speech understanding model that classifies key interactional events directly from audio. |
Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing literature has focused on pretrainer-based text-driven brain encoding models . however, few studies have explored the efficacy of task-specific learning of Transformers . |
| Approach: | They propose to use ten popular natural language processing tasks to learn Transformer representations for predicting brain responses. |
| Outcome: | The proposed model predicts brain activity across the whole brain. |
ScanEZ: Integrating Cognitive Models with Self-Supervised Learning for Spatiotemporal Scanpath Prediction (2025.acl-short)
Copied to clipboard
| Challenge: | ScanEZ framework provides a framework for predicting scanpaths during reading . masked modeling of eye movements and cognitive model simulations are used to kick-start training. |
| Approach: | They propose a framework for self-supervised learning that models scanpaths using synthetic data and a 3-D gaze objective inspired bymasked language modeling. |
| Outcome: | The proposed framework achieves state-of-the-art results on established datasets and is portable across different conditions. |