Papers with navigation
Representation, Learning and Reasoning on Spatial Language for Downstream NLP Tasks (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we discuss the cutting-edge research results and existing challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Approach: | This tutorial presents cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Outcome: | This paper reviews the cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models rely on rigid, hand-crafted rules to model nuanced behavior in urban environments. |
| Approach: | They propose an urban simulator that generates realistic daily schedules using a recursive value-driven approach that balances mandatory activities, personal habits, and situational factors. |
| Outcome: | The proposed urban simulator exhibits closer alignment with real humans than previous work. |
NavA3: Understanding Any Instruction, Navigating Anywhere, Finding Anything (2026.acl-long)
Copied to clipboard
Lingfeng Zhang, Xiaoshuai Hao, Yingbo Tang, Haoxiang Fu, Xinyu Zheng, Pengwei Wang, Zhongyuan Wang, Wenbo Ding, Shanghang Zhang
| Challenge: | Existing embodied navigation methods struggle with such tasks due to their limitations in comprehending high-level human instructions and localizing objects with an open vocabulary. |
| Approach: | They propose a hierarchical framework for long-horizon navigation that integrates human instructions with 3D scene views. |
| Outcome: | The proposed model achieves SOTA results and can complete long-horizon navigation tasks across different robot embodiments in real-world environments. |
LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view. |
| Approach: | They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions. |
| Outcome: | The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations. |
SCAN: Structured Capability Assessment and Navigation for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research has focused on approximating model rankings, but such benchmarks fail to provide users and developers with a comprehensive and fine-grained understanding of a specific model’s capabilities. |
| Approach: | They propose a framework that enables detailed characterization of LLM capabilities through comprehensive and fine-grained evaluation. |
| Outcome: | The proposed framework enables detailed characterization of large language models through comprehensive and fine-grained evaluation. |
Can LLMs Learn to Map the World from Local Descriptions? (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated strong capabilities in tasks such as code generation and mathematical reasoning. |
| Approach: | They investigate whether large language models can construct coherent global spatial cognition by integrating fragmented relational descriptions. |
| Outcome: | The proposed models can generalize to unseen spatial relationships and exhibit latent representations aligned with real-world spatial distributions. |
A Web Portal about the State of the Art of NLP Tasks in Spanish (2024.lrec-main)
Copied to clipboard
Enrique Amigó, Jorge Carrillo-de-Albornoz, Andrés Fernández, Julio Gonzalo, Guillermo Marco, Roser Morante, Laura Plaza, Jacobo Pedrosa
| Challenge: | a web portal has been created with information about the state of the art of natural language processing tasks in Spanish. |
| Approach: | They propose a web portal that provides information about the state of the art of natural language processing tasks in Spanish. |
| Outcome: | The portal provides information about forums, competitions, tasks and datasets in Spanish that would otherwise be spread in multiple articles and web sites. |
Behavior Cloned Transformers are Neurosymbolic Reasoners (2023.eacl-main)
Copied to clipboard
| Challenge: | In injecting actions from symbolic modules into the action space of a behavior cloned transformer agent increases performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . contemporary agents struggle on tasks such as navigation, admetic and other tasks that humans make use of external tools. |
| Approach: | They propose to inject actions from symbolic modules into the action space of a behavior cloned transformer agent to increase performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by an average of 22% . |
| Outcome: | The proposed method improves performance on four text game benchmarks that test arithmetic, navigation, sorting, and common sense reasoning by 22%, allowing an agent to reach the highest possible performance on unseen games. |
Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning (2025.findings-emnlp)
Copied to clipboard
Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, Jinhua Zhao
| Challenge: | Currently, vision-language models excel in many downstream tasks but struggle with spatial reasoning, which is crucial for navigation and interaction with physical environments. |
| Approach: | They propose a framework that generates synthetic data to provide targeted supervision for VLMs across these basic spatial capabilities. |
| Outcome: | The proposed framework disentangles 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. |
Sub-Instruction Aware Vision-and-Language Navigation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite significant advances, few previous works are able to fully utilize the strong correspondence between visual and textual sequences. |
| Approach: | They propose to provide agents with fine-grained annotations during training and provide them with sub-instructions and their corresponding paths. |
| Outcome: | The proposed method improves the performance of four state-of-the-art agents in a room-to-room (R2R) benchmark dataset. |
Understanding Spatial Relations through Multiple Modalities (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on common sense reasoning and understanding of spatial relations is limited. |
| Approach: | They propose a spatial model that uses both textual and visual information to predict spatial relations between two entities in an image. |
| Outcome: | The proposed model improves prediction accuracy and coverage and deals with unseen subjects, objects and relations. |
Beyond Blind Following: Evaluating Robustness of LLM Agents under Imperfect Guidance (2026.eacl-long)
Copied to clipboard
Yao Fu, Ran Qiu, Xinhe Wang, Jacob Sansom, Sathvika Ayyappa Prabhu, Huijie Tang, Jaekyeom Kim, Sungryull Sohn, Honglak Lee
| Challenge: | Large language models (LLMs) have shown strong capabilities as task-solving agents across interactive domains, but in complex environments, auxiliary guidance may be imperfect. |
| Approach: | They propose a benchmark to measure the robustness of large language models under imperfect guidance. |
| Outcome: | The proposed benchmark compared LLM agents in navigation, cooking, and gaming in a variety of environments with auxiliary guidance and noisy or underspecified instructions extracted from demonstrations. |
Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on Vision-and-Language Navigation (VLN) tasks do not measure how much of a language instruction the agent is able to follow. |
| Approach: | They propose a language-aligned supervision scheme that measures the number of sub-instructions the agent has completed during navigation. |
| Outcome: | The proposed method is based on the previous work on the Vision-and-Language Navigation task, which assumes a discrete navigation graph (navgraph) but not on the current work. |
Constructing Indonesian-English Travelogue Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | low-resource language research often hampered due to under-representation of how it is being used in reality. |
| Approach: | They propose to use a dataset comprising both Indonesian and English from personal travelogue articles . they used named and nominal expressions of four entity types related to travel . |
| Outcome: | The proposed dataset is more representative of how Indonesian language is being used in reality. |
ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments (2020.findings-emnlp)
Copied to clipboard
| Challenge: | embodied agents are expected to perform specific tasks after reaching the destination . a novel vision-and-language navigation task is designed to support this task . |
| Approach: | They combine vision-and-language navigation, assembling objects and object referring expression comprehension to create a joint navigation-and assembly task. |
| Outcome: | The proposed task is based on vision-and-language navigation and assembly . it uses human-written navigation and assembling instructions and ground truth trajectories . the large model-human performance gap shows that the task is challenging and wide scope for future work. |
Diagnosing Vision Language Models’ Perception by Leveraging Human Methods for Color Vision Deficiencies (2026.eacl-long)
Copied to clipboard
| Challenge: | Large-scale Vision-Language Models (LVLMs) are being deployed in real-world settings that require visual inference. |
| Approach: | They evaluate LVLMs' ability to account for variation in color perception using the Ishihara Test. |
| Outcome: | The proposed models fail to reproduce the perceptual outcomes experienced by affected individuals and default to normative color perception. |
DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on cross-modal attention at the fusion stage, but modality features generated by disparate uni-encoders reside in their own spaces, leading to a decline in the quality of cross-modulation and decision-making. |
| Approach: | They propose a framework to align navigation-related modalities before fusion by cross-modal contrastive learning. |
| Outcome: | The proposed framework integrates with the majority of existing models, resulting in improved navigation performance on various VLN benchmarks, including R2R, R4R, and CVDN. |
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM (2025.findings-acl)
Copied to clipboard
| Challenge: | High-performance vision-and-language navigation models require large amounts of training data, the high cost of manual annotating has seriously hindered this field. |
| Approach: | They propose a retrieval-augmented generation framework that generates user demand instructions for vision-and-language navigation. |
| Outcome: | The proposed model achieves SOTA performance on the REVERIE benchmark. |
ScreenQA: Large-Scale Question-Answer Pairs Over Mobile App Screenshots (2025.naacl-long)
Copied to clipboard
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Carbune, Jason Lin, Maria Wang, Yun Zhu, Jindong Chen
| Challenge: | Existing screen datasets focus on low-level structural and component understanding or on a much higher-level composite task such as navigation and task completion for autonomous agents. |
| Approach: | They propose to annotate 86k question-answer pairs over the RICO dataset to benchmark screen content understanding. |
| Outcome: | The proposed dataset covers full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. |
Emotags: Computer-Assisted Verbal Labelling of Expressive Audiovisual Utterances for Expressive Multimodal TTS (2024.lrec-main)
Copied to clipboard
| Challenge: | We show that ascribing verbal descriptions to expressive audiovisual utterances is efficient and efficient. |
| Approach: | They propose a web app for ascribing verbal descriptions to expressive audiovisual utterances. |
| Outcome: | The proposed system can be deployed at a large scale to efficiently collect relevant verbal descriptions. |
Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work on visual-grounded navigation has focused on indoor scenarios with sharp drops in performance when testing on unseen data. |
| Approach: | They focus on visual agent navigation in outdoor scenarios with panorama images . they find that most gain in outdoor VLN on unseen data is due to specific features . |
| Outcome: | The results show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments. |
Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show remarkable success in end-to-end task-oriented dialog systems . however, most models rely on large training data, which is difficult to scalable for new domains with limited labeled data. |
| Approach: | They propose a shared-private network which exploits the relevance between the target domain and each domain. |
| Outcome: | The proposed model outperforms existing methods on multi-domain dialogue by 13.9% on average. |
Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-and-Language Navigation (VLN) is a subfield of embodied AI that integrates natural language understanding, visual perception, and sequential decision-making to allow autonomous agents to navigate and interact within visual environments. |
| Approach: | They propose a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents. |
| Outcome: | The proposed framework decomposes navigation into atomic skills handled by a specialized agent. |
CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space (2025.emnlp-main)
Copied to clipboard
Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, Jincai Huang
| Challenge: | Embodied Question Answering (EQA) tasks are primarily focused on indoor environments, leaving the complexities of urban settings unexplored. |
| Approach: | They propose a task where an embodied agent answers open-vocabulary questions in dynamic city spaces. |
| Outcome: | The proposed agent achieves 60.7% of human-level answering accuracy compared to baselines . the proposed agent outperforms existing agents in open-ended city spaces . |
Refer360∘: A Referring Expression Recognition Dataset in 360∘ Images (2020.acl-main)
Copied to clipboard
| Challenge: | Refer360° is a large-scale referring expression recognition dataset consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes. |
| Approach: | They propose a large-scale referring expression recognition dataset, Refer360°, consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes. |
| Outcome: | The proposed dataset contains 17,137 instruction sequences and ground-truth actions for referring expression recognition in 360° scenes. |
Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot LLM-based Vision-and-Language Navigation agents either encode images as textual scene descriptions, potentially oversimplifying visual details, or process raw image inputs, which can fail to capture abstract semantics required for high-level reasoning. |
| Approach: | They propose to integrate large language models into embodied AI models by incorporating textual descriptions that facilitate analogical reasoning across images from multiple perspectives. |
| Outcome: | The proposed approach improves the agent’s contextual understanding on the R2R dataset, showing that it can make better decisions based on the LLMs. |
GUITester: Enabling GUI Agents for Exploratory Defect Discovery (2026.findings-acl)
Copied to clipboard
| Challenge: | Exploratory GUI testing is essential for software quality but suffers from high manual costs. |
| Approach: | They propose a framework that decouples navigation from verification via two modules . they propose 143 tasks and a GUITestBench benchmark that features 26 defects . |
| Outcome: | The proposed framework outperforms state-of-the-art benchmarks in 143 tasks and 26 defects. |
UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces (2025.acl-long)
Copied to clipboard
Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, Yong Li
| Challenge: | Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban aerial spaces remain to be explored. |
| Approach: | They propose a benchmark to evaluate whether large multimodal models can process continuous first-person visual observations like humans. |
| Outcome: | The proposed model can process first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. |
NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Graphical User Interface (GUI) and embodied navigation have driven progress, yet these domains have largely evolved in isolation, with disparate datasets and training paradigms. |
| Approach: | They propose a visual-target trajectory collection pipeline that generates trajectories for GUI and embodied tasks using a single formulation. |
| Outcome: | The proposed agent outperforms state-of-the-art agents in GUI navigation, spatial affordance prediction, and embodied navigation. |
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |