Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing metrics for vision-and-language navigation focus on goal completion rather than the sequence of actions corresponding to the instructions. |
| Approach: | They propose to use a room-to-room dataset to measure the length of instruction followed by agents. |
| Outcome: | The proposed metric outperforms existing metrics for Room-to-Room tasks because it is direct-to goal shortest. |
Similar Papers
Sub-Instruction Aware Vision-and-Language Navigation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite significant advances, few previous works are able to fully utilize the strong correspondence between visual and textual sequences. |
| Approach: | They propose to provide agents with fine-grained annotations during training and provide them with sub-instructions and their corresponding paths. |
| Outcome: | The proposed method improves the performance of four state-of-the-art agents in a room-to-room (R2R) benchmark dataset. |
VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Vision-and-Language Navigation benchmarks assume instructions are feasible and the referenced target exists. |
| Approach: | They propose a benchmark with false-premise instructions where the target is absent . they propose supervised room-level navigation with LLM/VLM-driven in-room exploration . |
| Outcome: | The proposed benchmark produces false-premise goals that are plausible but factually incorrect . ROAM achieves the best REV-SPL among compared methods, while baselines often under-explore and terminate prematurely under unreliable instructions. |
Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on Vision-and-Language Navigation (VLN) tasks do not measure how much of a language instruction the agent is able to follow. |
| Approach: | They propose a language-aligned supervision scheme that measures the number of sub-instructions the agent has completed during navigation. |
| Outcome: | The proposed method is based on the previous work on the Vision-and-Language Navigation task, which assumes a discrete navigation graph (navgraph) but not on the current work. |
Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents. |
| Approach: | They propose a solution that combines visual and linguistic features to enable VLN . they propose augmentation of the training data to fill the gap in missing patterns . |
| Outcome: | The proposed solution fills the gap in missing patterns of training data. |
VLN-Trans: Translator for the Vision and Language Navigation Agent (2023.acl-long)
Copied to clipboard
| Challenge: | We observe two kinds of instructions that make the grounding in the vision-and-language navigation task quite challenging. |
| Approach: | They propose to use a translator module to convert instructions into easy-to-follow sub-instruction representations at each step. |
| Outcome: | The proposed model is based on a Room2Room (R2R), Room4room (R4R), and Room2room Last (R1R-Last) datasets and achieves state-of-the-art results on multiple benchmarks. |
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Room-Across-Room (RxR) is a vision-and-language navigation dataset that addresses gaps in existing ones by addressing known biases in paths and eliciting more references to visible entities. |
| Approach: | They introduce a new Vision-and-Language Navigation (VLN) dataset that addresses biases in paths and elicits more references to visible entities. |
| Outcome: | The proposed model learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. |
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Using multilingual instructions to learn a better cross-lingual representation is challenging for multilingual agents. |
| Approach: | They propose to use multilingual instructions to learn a shared cross-lingual language representation for the three languages in a Room-Across-Room dataset. |
| Outcome: | The proposed model improves on the room-Across-room and vision-and-dialogue navigation tasks by maximizing similarity between semantically aligned image pairs from different environments. |
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing models that ground language into visual appearance and route structure are outperforming their visual counterparts in unseen new environments. |
| Approach: | They propose to decompose the grounding procedure into a set of expert models with access to different modalities and ensemble them at prediction time. |
| Outcome: | The proposed model outperforms models with only route structure and visual features on the benchmark Room-to-Room dataset. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a subfield of embodied AI that integrates natural language understanding, visual perception, and sequential decision-making to allow autonomous agents to navigate and interact within visual environments. |
| Approach: | They propose a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents. |
| Outcome: | The proposed framework decomposes navigation into atomic skills handled by a specialized agent. |