MULTICOLLAB: A Multimodal Corpus of Dialogues for Analyzing Collaboration and Frustration in Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to study complex emotions when a speaker collaborates with a partner are limited. |
| Approach: | They propose to fuse a multimodal dialogue resource with transcribed speech and eye gaze data to create a highly multimodal corpus. |
| Outcome: | The proposed model improves classification accuracy by 21% over baseline using sensor and speech data in 4.5 seconds. |
Similar Papers
FUSE - FrUstration and Surprise Expressions: A Subtle Emotional Multimodal Language Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | elicitation of frustration and surprise are understudied for emotion modeling in language, but are difficult to characterize. |
| Approach: | They propose a multimodal corpus for expressive task-based spoken language and dialogue focused on frustration and surprise, which are understudied for emotion modeling in language. |
| Outcome: | The proposed corpus provides both individual and dyadic multimodally grounded language. |
Modeling Collaborative Multimodal Behavior in Group Dialogues: The MULTISIMO Corpus (L18-1)
Copied to clipboard
| Challenge: | a corpus of human-computer interactions recorded in multiple modalities is being developed to study and model collaborative aspects of multimodal behavior in groups. |
| Approach: | They propose to use a multimodal corpus to investigate collaborative aspects of multimodal behavior in groups that perform simple tasks. |
| Outcome: | The proposed corpus is designed for public release and includes survey materials, personality tests and experience assessment questionnaires filled in by all participants. |
Construction and Analysis of a Multimodal Chat-talk Corpus for Dialog Systems Considering Interpersonal Closeness (2020.lrec-1)
Copied to clipboard
| Challenge: | a large-scale multimodal dialog corpus is needed to accelerate research on dialog systems that can handle social signals and verbal information. |
| Approach: | They construct a multimodal dialog corpus focusing on the relationship between speakers and 19 pairs of participants. |
| Outcome: | The proposed system is based on a multimodal dialog corpus of 19,303 utterances (10 hours) from 19 pairs of participants. |
The AICO Multimodal Corpus – Data Collection and Preliminary Analyses (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on human multimodal behaviour in interactions with a human or a robot partner are limited. |
| Approach: | They describe the first explorative research on the AICO Multimodal Corpus, which contains eye-gaze, Kinect, and video recordings of human-robot and human-human interactions. |
| Outcome: | The AICO Multimodal Corpus contains eye-gaze, Kinect, and video recordings of human-robot and human-human interactions. |
In Search of the Lost Arch in Dialogue: A Dependency Dialogue Acts Corpus for Multi-Party Dialogues (2025.findings-acl)
Copied to clipboard
Jon Cai, Brendan King, Peyton Cameron, Susan Windisch Brown, Miriam Eckert, Dananjay Srinivas, George Arthur Baker, V Kate Everson, Martha Palmer, James Martin, Jeffrey Flanigan
| Challenge: | Understanding speaker intentions remains a challenge in NLP . a number of corpora annotated using theoretical frameworks of dialogue focus on utterance-level labeling of speaker intent, missing wider context, or the rhetorical structure of a dialogue. |
| Approach: | They propose to annotate a corpus of 33 dialogues and over 9,000 utterance units using the Dependency Dialogue Acts framework. |
| Outcome: | The proposed corpus spans four genres of multi-party conversations from different modalities. |
A Large Scale Speech Sentiment Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing corpus for sentiment analysis uses text inputs, but voice inputs are becoming more important as smart assistants and mobile voice control become more prevalent. |
| Approach: | They propose to extend the Switchboard-1 Telephone Speech Corpus by adding sentiment labels from 3 different human annotators for every transcript segment. |
| Outcome: | The proposed corpus contains 49500 labeled speech segments covering 140 hours of audio. |
A Multimodal Corpus for Mutual Gaze and Joint Attention in Multiparty Situated Interaction (L18-1)
Copied to clipboard
Dimosthenis Kontogiorgos, Vanya Avramova, Simon Alexanderson, Patrik Jonell, Catharine Oertel, Jonas Beskow, Gabriel Skantze, Joakim Gustafson
| Challenge: | Using a multisensory setup, we capture speech, eye gaze and gesture data and investigate four different types of social gaze: referential gaze, joint attention, mutual gaze and gaze aversion by both perspectives of a speaker and a listener. |
| Approach: | They present a corpus of situated interaction where participants collaborated on moving virtual objects on a large touch screen. |
| Outcome: | The authors capture speech, eye gaze and gesture data using a multisensory setup and analysed the groups' referential eye-gaze with respect to the referent object. |
Multimodal Behaviour in an Online Environment: The GEHM Zoom Corpus Collection (2024.lrec-main)
Copied to clipboard
| Challenge: | Several studies have discussed pros and cons of videoconferencing for group meetings, international conference organisation and teaching. |
| Approach: | They propose to use 12 video recordings of Zoom meetings held in English by an international group of researchers from September 2021 to March 2023 to study group communication in a reallife setting. |
| Outcome: | The proposed corpus was developed under the auspices of the international network on Gesture and Head Movement in Language (GEHM) it shows that the participants' speech transcription and visual keypoint values can be visualised to see how gestural behaviour supports feedback words during the interaction. |
Multimodal large language models for inclusive collaboration learning tasks (2022.naacl-srw)
Copied to clipboard
| Challenge: | This project leverages advances in multimodal large language models to build an inclusive collaboration feedback loop for participants developing general collaboration skills. |
| Approach: | They propose to integrate advances in multimodal large language models into downstream tasks such as the learning analytics feedback loop. |
| Outcome: | The proposed model will be used to detect, model, and feedback participants developing general collaboration skills. |
MEISD: A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations (2020.coling-main)
Copied to clipboard
| Challenge: | Emotion and sentiment classification in dialogues has gained popularity in recent times . a number of datasets are imbalanced in representing different emotions and consist of an only single emotion. |
| Approach: | They propose to use a dataset to analyze emotions and sentiments in dialogues . they use text, audio and video to identify the correct emotions with the appropriate intensity and sentiment in an utterance of a dialogue . |
| Outcome: | The proposed datasets are balanced in representing different emotions and consist of only one emotion. |