| Challenge: | Existing visual dialog methods use RNN to encode the dialog history as a vector representation . a new method for video dialog is proposed, which progressively updates query information based on dialog history and video content until the agent think the information is sufficient and unambiguous. |
| Approach: | They propose a method which progressively updates query information based on dialog history and video content until the agent thinks it is sufficient and unambiguous. |
| Outcome: | The proposed method can be used to infer video dialog answers on large-scale datasets. |
Similar Papers
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing work on video-grounded dialogue systems is limited by feature space and semantic information. |
| Approach: | They propose multimodal transformer networks to encode videos and incorporate information from different modalities. |
| Outcome: | The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. |
Dialog Generation Using Multi-Turn Reasoning Neural Networks (N18-1)
Copied to clipboard
| Challenge: | Existing methods for dialog generation are limited and short at generalization. |
| Approach: | They propose a generalizable dialog generation approach that adapts multi-turn reasoning to generate responses by taking current conversation session context as a document and current query as 'question' they separate the single memory used for document comprehension into different groups for speaker-specific topic and opinion embedding. |
| Outcome: | Experiments on Japanese 10-sentence (5-round) conversation modeling show that multi-turn reasoning can produce more diverse and acceptable responses than state-of-the-art single-turn and non-reasoning baselines. |
Multi-Scale Progressive Attention Network for Video Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Experimental evaluations on three benchmarks: TGIF-QA, MSVD-QA and MSRVTT-QA show our method has achieved state-of-the-art performance. |
| Approach: | They propose a multi-scale progressive attention network to fuse visual and text information. |
| Outcome: | The proposed method achieves state-of-the-art on three benchmarks: TGIF-QA, MSVD-QA and MSRVTT-QA. |
Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for video-grounded dialogue generation do not allow information from different modalities to complement each other. |
| Approach: | They propose a video-grounded dialogue generation model that integrates video data into pre-trained language models to allow information from different modalities to complement each other. |
| Outcome: | The proposed model outperforms state-of-the-art models on automatic and human evaluations on two public datasets. |
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances in natural language processing and computer vision have made significant progress in artificial intelligence (AI). |
| Approach: | They propose Motion-Appearance Synergistic Networks which embed cross-modal features grounded on motion and appearance information and selectively utilize them depending on the question’s intentions. |
| Outcome: | The proposed network achieves state-of-the-art on the TGIF-QA and MSVD-QA datasets and qualitatively analyzes the results. |
Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for temporal sentence localization in videos focus on visual content, but they are insufficient to model complex video contents. |
| Approach: | They propose a deep rectification-modulation network to correct attention misalignment . they use sentence information to capture frame-to-frame relation . |
| Outcome: | The proposed method achieves state-of-the-art performance on three public datasets. |
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent work has focused on learning architectures with large memories capable of storing external knowledge. |
| Approach: | They propose a method to augment generative Transformer neural networks with information fetching modules. |
| Outcome: | The proposed approach improves performance in generative dialog modeling . external knowledge is retrieved from Wikipedia, images, and human-written dialog utterances . |
Dual Dynamic Memory Network for End-to-End Multi-turn Task-oriented Dialog Systems (2020.coling-main)
Copied to clipboard
| Challenge: | Existing task-oriented dialog systems struggle to dynamically model long dialog context for interactions and effectively incorporate knowledge base (KB) information into dialog generation. |
| Approach: | They propose a dual dynamic memory network for multi-turn dialog generation . the model dynamically expands the dialog memory turn by turn and keeps track of dialog history . |
| Outcome: | The proposed model outperforms baseline models on three benchmark datasets on human evaluation and automatic evaluation. |
On Pursuit of Designing Multi-modal Transformer for Video Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video grounding are not end-to-end, i.e., they rely on time-consuming post-processing steps to refine predictions. |
| Approach: | They propose an end-to-end multi-modal Transformer model that uses two encoders and a cross-modal decoder for grounding prediction. |
| Outcome: | The proposed model is 4.9% faster than existing models and is based on a set of encodings and decoders. |
Hierarchical Transformer for Task Oriented Dialog Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models for dialog generation are challenging to train using the standard Seq2Seq models. |
| Approach: | They propose a framework for Hierarchical Transformer Encoders that can be morphed into any hierarchical transformer by using specially designed attention masks and positional encodings. |
| Outcome: | The proposed framework can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. |