| Challenge: | Aerial visionand-dialling navigation (AVDN) is a new approach to autonomous drones that can converse with humans and follow natural language commands to complete tasks. |
| Approach: | They propose to use Aerial Visionand-Dialog Navigation (AVDN) to navigate a drone via natural language conversation by collecting a dataset of over 3k recorded navigation trajectories with asynchronous human-human dialogs between commanders and followers. |
| Outcome: | The proposed system can converse with humans and follow natural language commands to fly to the expected destination. |
Similar Papers
Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for vision-and-dialog navigation are limited and do not adapt to domain changes. |
| Approach: | They propose a problem where an agent computes dialog-navigation policies from trial and error. |
| Outcome: | The proposed agent outperforms baselines in success rate in photo-realistic simulations. |
GesNavi: Gesture-guided Outdoor Vision-and-Language Navigation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing datasets for outdoor Vision-and-Language Navigation (VLN) tasks do not include verbal instructions for communicating with mobility. |
| Approach: | They propose a dataset for gesture-guided outdoor VLN instructions with demonstrative expressions that incorporates gestures and linguistic commands. |
| Outcome: | The proposed datasets are compared against existing datasets and analysed in detail. |
VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator (2022.findings-acl)
Copied to clipboard
Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gokhan Tur, Devi Parikh, Dilek Hakkani-Tur
| Challenge: | Interactive robots navigating photo-realistic environments need to be trained to handle dynamic nature of dialogue and vision-and-language navigation (VLN). |
| Approach: | They propose a Transformer-based multi-modal navigator that is better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). |
| Outcome: | The proposed model is trained to identify and associate object-level concepts and semantics between the environment and dialogue history and identify when to interact vs. navigate via imitation learning of a binary classification head. |
FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Hengxing Cai, Jinhan Dong, Jingjun Tan, Jingcheng Deng, Sihang Li, Zhifeng Gao, Haidong Wang, Zicheng Su, Agachai Sumalee, Renxin Zhong
| Challenge: | Existing methods for vision-and-language navigation struggle with insufficient multimodal fusion, weak generalization, and poor interpretability. |
| Approach: | They propose a framework for UAV vision-and-language navigation that integrates natural language instructions with visual observations to improve multimodal fusion and interpretability. |
| Outcome: | The proposed framework achieves state-of-the-art performance across all scenarios, with a 9.22% higher success rate than the strongest baseline in unseen environments. |
NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue (2021.emnlp-main)
Copied to clipboard
| Challenge: | Vision-and-Dialogue Navigation is one of the tasks that evaluate the agent’s ability to interact with humans for assistance and navigate based on natural language responses. |
| Approach: | They propose a vision-and-dialogue navigation task which evaluates the agent's ability to interact with humans and navigate based on natural language responses. |
| Outcome: | The proposed model performs well on the Navigation from Dialogue History task, but it is not evaluated by the primary metric Goal Progress. |
Connecting Language and Vision to Actions (P18-5)
Copied to clipboard
| Challenge: | Recent advances in language and vision have made incredible progress in describing images and interacting with visual content in a physical or embodied environment. |
| Approach: | This tutorial will provide an overview of the growing number of multimodal tasks and datasets that combine textual and visual understanding. |
| Outcome: | This tutorial will review the state-of-the-art approaches to selected tasks such as image captioning, visual question answering and visual dialog. |
AV-Dialog: Spoken Dialogue Models with Audio-Visual Input (2026.acl-long)
Copied to clipboard
| Challenge: | AV-Dialog uses audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. |
| Approach: | They propose a multimodal dialog framework that uses both audio and visual cues to track the target speaker. |
| Outcome: | AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction and human-rated dialogue quality. |
Transformer-based Localization from Embodied Dialog with Large-scale Pre-training (2022.aacl-short)
Copied to clipboard
| Challenge: | Observer agent and Locator agents communicate via natural language dialog without access to the Observers view of the scene. |
| Approach: | They propose a graph-based scene representation that is more effective than top-down 2D maps used in previous works. |
| Outcome: | The proposed model outperforms top-down 2D maps and is more accurate than previous baselines. |
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams (2023.acl-long)
Copied to clipboard
Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, Seungwhan Moon
| Challenge: | Existing models lack a large-scale benchmark to capture user–assistant interactions . et al., 2022: 145-160. |
| Approach: | They propose a video-grounded task-oriented dialog dataset that captures real-world AI-assisted user scenarios in VR. |
| Outcome: | The proposed dataset captures real-world AI-assisted user scenarios in VR. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |