| Challenge: | Current dialogue systems focus more on textual and speech context knowledge and are usually based on two speakers. |
| Approach: | They propose to use live soccer game videos and Twitch.tv chats to develop visual-grounded dialogue models. |
| Outcome: | The proposed model can generate relevant temporal and spatial event language from live video and chat history while also being relevant to chat history. |
Similar Papers
VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions (2023.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for video-grounded dialogues neglect the intrinsic attributes of multimodal dialogues, such as scene and topic transitions. |
| Approach: | They propose to use a large scale video-grounded scene&topic AwaRe dialogue dataset to study video-based dialogue understanding. |
| Outcome: | The proposed dataset shows that multimodal information and segments are important in video-grounded dialogue understanding and generation. |
Video-Grounded Dialogues with Pretrained Generation Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown success in improving downstream NLP tasks . pre-tuned models capture textual dependencies in text data of rich semantics . |
| Approach: | They propose a framework for improving video-grounded dialogue by extending GPT-2 models . they propose to combine visual and textual representation into a structured sequence . |
| Outcome: | The proposed framework improves audio-visual scene-aware dialogues benchmark on AVSD . it is based on a large pre-trained GPT-2 network and can generate natural responses . |
Event-Content-Oriented Dialogue Generation in Short Video (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing multi-modal dialogue models are limited to incapacity of reading visual information and multi-dimensional interactions. |
| Approach: | They propose a novel event-oriented video-dialogue dataset called SportsVD to overcome these challenges by generating human-like response according to event contents in the video and related external knowledge. |
| Outcome: | The proposed method outperforms existing methods on SportsVD and other baselines under several automatic metrics. |
Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for video-grounded dialogue generation do not allow information from different modalities to complement each other. |
| Approach: | They propose a video-grounded dialogue generation model that integrates video data into pre-trained language models to allow information from different modalities to complement each other. |
| Outcome: | The proposed model outperforms state-of-the-art models on automatic and human evaluations on two public datasets. |
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks do not have enough annotations to analyze video-grounded dialogue systems and understand their capabilities and limitations in isolation. |
| Approach: | They present a Diagnostic Dataset for Video-grounded dialogue with minimal biases and detailed annotations for the different types of reasoning over the spatio-temporal space of video. |
| Outcome: | The proposed system is based on 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video. |
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing work on video-grounded dialogue systems is limited by feature space and semantic information. |
| Approach: | They propose multimodal transformer networks to encode videos and incorporate information from different modalities. |
| Outcome: | The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. |
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | a surge of deep learning applications for video understanding have led to major advancements in video-related tasks. |
| Approach: | They propose a multimodal video-based conversation model that merges a video-adapted visual encoder with an LLM and a dataset that is easily scalable and robust to label noise. |
| Outcome: | The proposed model can understand and generate detailed conversations about videos. |
Image-Chat: Engaging Grounded Conversations (2020.acl-main)
Copied to clipboard
| Challenge: | In order for machines to communicate with humans, they must understand the natural things that humans say about the world they live in and respond in kind. |
| Approach: | They propose to fuse a set of neural architectures using image and text representations to achieve this goal. |
| Outcome: | The proposed model performs well on the Image-Chat task and humans prefer it 47.7% of the time. |
A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge. |
| Approach: | They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games. |
| Outcome: | The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players. |
The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue (P19-1)
Copied to clipboard
| Challenge: | Using the PhotoBook dataset, we investigate shared dialogue history accumulating during conversation . human interlocutors are known to collaboratively establish a shared repository of mutual information during a conversation - this common ground is then used to optimise understanding and communication efficiency. |
| Approach: | They propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context and previously established referring expressions. |
| Outcome: | The proposed model takes into account shared information accumulated in a reference chain and is important to resolve later descriptions. |