| Challenge: | Vision-Language Models (VLMs) have shown promise as web agents, yet their planning has been overlooked. |
| Approach: | They propose to examine VLMs’ ability to understand temporal relationships within web contexts and assess plans of actions across diverse scenarios. |
| Outcome: | The proposed models exhibit limited performance in the above skills and are not reliable to function as web agents. |
Similar Papers
Measuring Progress in Fine-grained Vision-and-Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | X-VLM models lack "fine-grained" understanding of relationships, verbs and numbers in images . pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language tasks . |
| Approach: | They investigate models that outperform other baselines on fine-grained data . they highlight importance of novel losses and rich data sources for learning fine-grain skills . |
| Outcome: | The proposed model outperforms baseline models on four fine-grained benchmarks . the model outpersforms other baseline models and even degrades performance . |
Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents. |
| Approach: | They propose a solution that combines visual and linguistic features to enable VLN . they propose augmentation of the training data to fill the gap in missing patterns . |
| Outcome: | The proposed solution fills the gap in missing patterns of training data. |
Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language Models have been shown to be highly capable but lacking basic visual understanding skills. |
| Approach: | They propose to examine the limitations of vision-language models on visual tasks by constructing a series of tests that probe which components of design may be lacking. |
| Outcome: | The proposed tests compare VLMs to other models on visual encoders, intermediate vision-language projection and LLM-decoder outputs. |
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on developing LLMs to automate complex planning tasks. |
| Approach: | They propose to provide a comprehensive overview of current LLM planners to fill this gap . they examine performance criteria including completeness, executability, optimality, representation, generalization, and efficiency . |
| Outcome: | The proposed survey examines performance criteria for LLM planners and highlights their strengths and weaknesses. |
MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding (2024.findings-emnlp)
Copied to clipboard
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Liujian Liujianfeng, Ang Li, Jian Luan, Bin Wang, Shuo Shang
| Challenge: | Recent mobile AI agents based on VLMs lack basic mobile capabilities due to their pre-trained nature. |
| Approach: | They propose a mobile AI agent based on VLMs that includes additional pre-training stages to enhance both intra- and inter-UI understanding. |
| Outcome: | The proposed model outperforms existing VLMs on the Chinese mobile dataset Mobile3M . |
Can LLMs Truly Plan? A Comprehensive Evaluation of Planning Capabilities (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing assessments of planning capabilities of large language models are limited to single-language or specific representation formats. |
| Approach: | a new benchmark is developed to assess the planning capabilities of large language models. |
| Outcome: | The Multi-Plan benchmark highlights performance disparities among models . language differences showed minimal impact, while mathematically structured representations improved accuracy . |
Creative Planning with Language Models: Practice, Evaluation and Applications (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial explores how planning has been learned and deployed in creative workflows . many human creative tasks involve extensive planning, and actions need to be taken . |
| Approach: | This tutorial explores how planning has been learned and deployed in creative workflows . authors discuss forward and backward learning approaches for planning in LLMs - and evaluation metrics tailored to latent plans . |
| Outcome: | This tutorial examines how planning has been learned and deployed in creative workflows . it discusses forward and backward learning approaches for planning in LLMs - evaluation metrics tailored to latent plans . |
Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus primarily on end-to-end success, offering limited insight into where failures arise. |
| Approach: | They propose a hierarchical planning framework that analyzes web agents across three layers . they show that structured Planning Domain Definition Language (PDDL) plans produce more concise and goal-directed strategies than natural language (NL) plans . |
| Outcome: | The proposed framework analyzes web agents across three layers to improve reasoning, grounding, and recovery. |
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning. |
| Approach: | They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions. |
| Outcome: | The proposed frameworks are used to support reliable off-the-shelf AP planners. |
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)
Copied to clipboard
Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu
| Challenge: | Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent. |
| Approach: | They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs. |
| Outcome: | The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models. |