The Niki and Julie Corpus: Collaborative Multimodal Dialogues between Humans, Robots, and Virtual Agents (L18-1)
Copied to clipboard
Ron Artstein, Jill Boberg, Alesia Gainer, Jonathan Gratch, Emmanuel Johnson, Anton Leuski, Gale Lucas, David Traum
| Challenge: | Niki and Julie corpus contains more than 600 dialogues between humans and robots . corpus includes audio and video recordings, results of ranking tasks, questionnaire responses . |
| Approach: | the corpus contains more than 600 dialogues between human participants and a robot . the dialogues are part of a collaborative item-ranking task designed to measure influence . |
| Outcome: | the corpus contains more than 600 dialogues between human participants and a robot or virtual agent . the dialogues contain conversational errors by the robot, which simulates typical of modern automated agents . |
Similar Papers
Modeling Collaborative Multimodal Behavior in Group Dialogues: The MULTISIMO Corpus (L18-1)
Copied to clipboard
| Challenge: | a corpus of human-computer interactions recorded in multiple modalities is being developed to study and model collaborative aspects of multimodal behavior in groups. |
| Approach: | They propose to use a multimodal corpus to investigate collaborative aspects of multimodal behavior in groups that perform simple tasks. |
| Outcome: | The proposed corpus is designed for public release and includes survey materials, personality tests and experience assessment questionnaires filled in by all participants. |
RDG-Map: A Multimodal Corpus of Pedagogical Human-Agent Spoken Interactions. (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of 209 spoken game dialogues between a human and a remote-controlled artificial agent is presented. |
| Approach: | They present a multimodal corpus of 209 spoken game dialogues between a human and a remote-controlled artificial agent. |
| Outcome: | The proposed corpus consists of 209 spoken game dialogues between a human and a remote-controlled artificial agent. |
An Information-Providing Closed-Domain Human-Agent Interaction Corpus (L18-1)
Copied to clipboard
| Challenge: | a human-agent interaction corpus is a corpus of conversations between a user and an embodied conversational agent operated by a wizard of oz . data collected to create a 'corpus' with unexpected situations, such as misunderstandings, false information, and interruptions. |
| Approach: | They propose a public corpus for Human-Agent Interaction where the agent is controlled by a Wizard of Oz. |
| Outcome: | The proposed corpus is based on 15 conversations between users and a wizard of Oz agent . the data are used to create a corpus with unexpected situations, such as misunderstandings, false information, and interruptions. |
Multimodal Corpus of Bidirectional Conversation of Human-human and Human-robot Interaction during fMRI Scanning (2020.lrec-1)
Copied to clipboard
Birgit Rauchbauer, Youssef Hmamouche, Brigitte Bigi, Laurent Prévot, Magalie Ochs, Thierry Chaminade
| Challenge: | a study of real-life bi-directional conversations combines multimodal corpus with neural, physiological and behavioral data. |
| Approach: | They propose a multimodal corpus derived from natural conversations . they used human-human interactions as a control condition . |
| Outcome: | The proposed corpus includes neural, physiological and behavioral data. |
MULTICOLLAB: A Multimodal Corpus of Dialogues for Analyzing Collaboration and Frustration in Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to study complex emotions when a speaker collaborates with a partner are limited. |
| Approach: | They propose to fuse a multimodal dialogue resource with transcribed speech and eye gaze data to create a highly multimodal corpus. |
| Outcome: | The proposed model improves classification accuracy by 21% over baseline using sensor and speech data in 4.5 seconds. |
FUSE - FrUstration and Surprise Expressions: A Subtle Emotional Multimodal Language Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | elicitation of frustration and surprise are understudied for emotion modeling in language, but are difficult to characterize. |
| Approach: | They propose a multimodal corpus for expressive task-based spoken language and dialogue focused on frustration and surprise, which are understudied for emotion modeling in language. |
| Outcome: | The proposed corpus provides both individual and dyadic multimodally grounded language. |
WikiConv: A Corpus of the Complete Conversational History of a Large Online Collaborative Community (D18-1)
Copied to clipboard
Yiqing Hua, Cristian Danescu-Niculescu-Mizil, Dario Taraborelli, Nithum Thain, Jeffery Sorensen, Lucas Dixon
| Challenge: | Compared to large-scale collections of conversations from social media, Wikipedia talk pages only capture a subset of all discussions and only accounts for the final form of each conversation. |
| Approach: | They propose to reconstruct a corpus that encompasses the complete history of conversations between Wikipedia contributors. |
| Outcome: | The proposed corpus extracts high quality data in both Chinese and English. |
Multimodal Large Language Models for Human-AI Interaction: Foundations, Agents, and Inclusive Applications (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Approach: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Outcome: | This tutorial covers foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
The AICO Multimodal Corpus – Data Collection and Preliminary Analyses (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on human multimodal behaviour in interactions with a human or a robot partner are limited. |
| Approach: | They describe the first explorative research on the AICO Multimodal Corpus, which contains eye-gaze, Kinect, and video recordings of human-robot and human-human interactions. |
| Outcome: | The AICO Multimodal Corpus contains eye-gaze, Kinect, and video recordings of human-robot and human-human interactions. |
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2020.acl-main)
Copied to clipboard
| Challenge: | a set of 12 tasks that measure if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, and perceive and converse about images. |
| Approach: | They propose a set of 12 tasks that measure if a conversational agent can communicate engagingly with personality and empathy . they use large dialogue datasets to multi-task and obtain state-of-the-art results . |
| Outcome: | The proposed model improves over a BERT pre-trained model on large dialogue datasets and provides state-of-the-art results on many of the tasks. |