Action Inference for Destination Prediction in Vision-and-Language Navigation (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing work on vision-and-language navigation focuses on spatial reasoning and semantic grounding of visual information, but there is still scope for improvement. |
| Approach: | They propose a VLN task of destination prediction for picking up a pedestrian that requires action inference from a crowd-sourced dataset. |
| Outcome: | The proposed model can reason about the effect of the next action and the next on the destination to a certain extent. |
Similar Papers
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions (2022.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a research topic that is gaining attention in the field of artificial intelligence. |
| Approach: | They propose to build an embodied agent that can communicate with humans in natural language and navigate in real 3D environments. |
| Outcome: | This paper reviews current studies in the emerging field of vision-and-language navigation . it highlights limitations and opportunities for future work . |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
GesNavi: Gesture-guided Outdoor Vision-and-Language Navigation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing datasets for outdoor Vision-and-Language Navigation (VLN) tasks do not include verbal instructions for communicating with mobility. |
| Approach: | They propose a dataset for gesture-guided outdoor VLN instructions with demonstrative expressions that incorporates gestures and linguistic commands. |
| Outcome: | The proposed datasets are compared against existing datasets and analysed in detail. |
Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on visual-grounded navigation has focused on indoor scenarios with sharp drops in performance when testing on unseen data. |
| Approach: | They focus on visual agent navigation in outdoor scenarios with panorama images . they find that most gain in outdoor VLN on unseen data is due to specific features . |
| Outcome: | The results show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments. |
Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents. |
| Approach: | They propose a solution that combines visual and linguistic features to enable VLN . they propose augmentation of the training data to fill the gap in missing patterns . |
| Outcome: | The proposed solution fills the gap in missing patterns of training data. |
Masked Path Modeling for Vision-and-Language Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | A major challenge in vision-and-language navigation is the limited available training data, which hinders the models’ ability to generalize effectively. |
| Approach: | They propose a masked path modeling objective that pretrains an agent using self-collected data for subsequent navigation tasks. |
| Outcome: | The proposed model pretrains an agent using self-collected data for subsequent navigation tasks eliminating the need for external tools. |
Robust Navigation with Language Pretraining and Stochastic Sampling (D19-1)
Copied to clipboard
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, Yejin Choi
| Challenge: | Existing methods to learn visual representations and action decoding schemes are limited to previously unseen instructions and environments. |
| Approach: | They propose a stochastic sampling scheme to reduce the gap between the expert actions in training and sampled actions in test to correct its own mistakes. |
| Outcome: | The proposed methods achieve 6% absolute gain over the previous best results on the Room-to-Room benchmark. |
Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models treat STOP as other actions, which leads to undesirable behaviors that the agent fails to stop at the destination. |
| Approach: | They propose a policy module that differentiates STOP from other actions . they propose 'learning to stop' module that can be used to train an agent to follow natural language instructions in real-world environments. |
| Outcome: | The proposed model outperforms the baseline on a challenging urban VLN dataset Touchdown by 6.89%. |
VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Vision-and-Language Navigation benchmarks assume instructions are feasible and the referenced target exists. |
| Approach: | They propose a benchmark with false-premise instructions where the target is absent . they propose supervised room-level navigation with LLM/VLM-driven in-room exploration . |
| Outcome: | The proposed benchmark produces false-premise goals that are plausible but factually incorrect . ROAM achieves the best REV-SPL among compared methods, while baselines often under-explore and terminate prematurely under unreliable instructions. |
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing models that ground language into visual appearance and route structure are outperforming their visual counterparts in unseen new environments. |
| Approach: | They propose to decompose the grounding procedure into a set of expert models with access to different modalities and ensemble them at prediction time. |
| Outcome: | The proposed model outperforms models with only route structure and visual features on the benchmark Room-to-Room dataset. |