Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation (2021.eacl-main)
Copied to clipboard
Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
| Challenge: | Outdoor vision-and-language navigation (VLN) tasks require visual grounding to generate correct actions. |
| Approach: | They propose a multimodal text style transfer learning approach to mitigate data scarcity in outdoor vision-and-language navigation tasks. |
| Outcome: | The proposed approach outperforms baseline models on the outdoor vision-and-language navigation task, improving task completion rate by 8.7% relative to the baseline models. |
Similar Papers
Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on visual-grounded navigation has focused on indoor scenarios with sharp drops in performance when testing on unseen data. |
| Approach: | They focus on visual agent navigation in outdoor scenarios with panorama images . they find that most gain in outdoor VLN on unseen data is due to specific features . |
| Outcome: | The results show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments. |
GesNavi: Gesture-guided Outdoor Vision-and-Language Navigation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing datasets for outdoor Vision-and-Language Navigation (VLN) tasks do not include verbal instructions for communicating with mobility. |
| Approach: | They propose a dataset for gesture-guided outdoor VLN instructions with demonstrative expressions that incorporates gestures and linguistic commands. |
| Outcome: | The proposed datasets are compared against existing datasets and analysed in detail. |
CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory (2025.acl-long)
Copied to clipboard
Weichen Zhang, Chen Gao, Shiquan Yu, Ruiying Peng, Baining Zhao, Qian Zhang, Jinqiang Cui, Xinlei Chen, Yong Li
| Challenge: | Existing ground VLN agents struggle in aerial VLLN due to the lack of predefined navigation graphs and the exponentially expanding action space in long-horizon exploration. |
| Approach: | They propose a large language model-empowered aerial VLN agent that decomposes the long-horizon task into sub-goals with different semantic levels. |
| Outcome: | The proposed method achieves state-of-the-art performance with significant improvement in continuous city environments. |
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Using multilingual instructions to learn a better cross-lingual representation is challenging for multilingual agents. |
| Approach: | They propose to use multilingual instructions to learn a shared cross-lingual language representation for the three languages in a Room-Across-Room dataset. |
| Outcome: | The proposed model improves on the room-Across-room and vision-and-dialogue navigation tasks by maximizing similarity between semantically aligned image pairs from different environments. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions (2022.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a research topic that is gaining attention in the field of artificial intelligence. |
| Approach: | They propose to build an embodied agent that can communicate with humans in natural language and navigate in real 3D environments. |
| Outcome: | This paper reviews current studies in the emerging field of vision-and-language navigation . it highlights limitations and opportunities for future work . |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)
Copied to clipboard
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
| Challenge: | Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English. |
| Approach: | They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. |
| Outcome: | The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting. |
Action Inference for Destination Prediction in Vision-and-Language Navigation (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing work on vision-and-language navigation focuses on spatial reasoning and semantic grounding of visual information, but there is still scope for improvement. |
| Approach: | They propose a VLN task of destination prediction for picking up a pedestrian that requires action inference from a crowd-sourced dataset. |
| Outcome: | The proposed model can reason about the effect of the next action and the next on the destination to a certain extent. |
Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents. |
| Approach: | They propose a solution that combines visual and linguistic features to enable VLN . they propose augmentation of the training data to fill the gap in missing patterns . |
| Outcome: | The proposed solution fills the gap in missing patterns of training data. |
Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a subfield of embodied AI that integrates natural language understanding, visual perception, and sequential decision-making to allow autonomous agents to navigate and interact within visual environments. |
| Approach: | They propose a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents. |
| Outcome: | The proposed framework decomposes navigation into atomic skills handled by a specialized agent. |