Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus primarily on end-to-end success, offering limited insight into where failures arise. |
| Approach: | They propose a hierarchical planning framework that analyzes web agents across three layers . they show that structured Planning Domain Definition Language (PDDL) plans produce more concise and goal-directed strategies than natural language (NL) plans . |
| Outcome: | The proposed framework analyzes web agents across three layers to improve reasoning, grounding, and recovery. |
Similar Papers
On the Limit of Language Models as Planning Formalizers (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models can create plans that are neither executable nor verifiable in grounded environments. |
| Approach: | They use Large Language Models to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). |
| Outcome: | The proposed model outperforms the models directly generating plans while being robust to lexical perturbation. |
Revealing the Barriers of Language Agents in Planning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show language agents lack human-level planning abilities . limitations and mechanisms to address them remain insufficiently understood . |
| Approach: | They apply a feature attribution study to identify key factors hindering agent planning . they identify the limited role of constraints and diminishing influence of questions . |
| Outcome: | The proposed model achieves 15.6% on a real-world planning benchmark. |
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning. |
| Approach: | They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions. |
| Outcome: | The proposed frameworks are used to support reliable off-the-shelf AP planners. |
Can LLMs Truly Plan? A Comprehensive Evaluation of Planning Capabilities (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing assessments of planning capabilities of large language models are limited to single-language or specific representation formats. |
| Approach: | a new benchmark is developed to assess the planning capabilities of large language models. |
| Outcome: | The Multi-Plan benchmark highlights performance disparities among models . language differences showed minimal impact, while mathematically structured representations improved accuracy . |
Personal Large Language Model Agents: A Case Study on Tailored Travel Planning (2024.emnlp-industry)
Copied to clipboard
Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj, Homa Fashandi, Kevin Ferreira, Chul Lee
| Challenge: | Large Language Models (LLMs) are becoming more autonomous and capable of handling real-world tasks through their access to tools, various planning strategies, and memory, referred to as LLM agents. |
| Approach: | They introduce a personalized version of TravelPlanner and establish baselines for personal LLM agents by comparing generic and personal plans. |
| Outcome: | The proposed model encapsulates user-related information, preferences, and personal concepts and provides baselines for personal LLM agents. |
Beyond Blind Following: Evaluating Robustness of LLM Agents under Imperfect Guidance (2026.eacl-long)
Copied to clipboard
Yao Fu, Ran Qiu, Xinhe Wang, Jacob Sansom, Sathvika Ayyappa Prabhu, Huijie Tang, Jaekyeom Kim, Sungryull Sohn, Honglak Lee
| Challenge: | Large language models (LLMs) have shown strong capabilities as task-solving agents across interactive domains, but in complex environments, auxiliary guidance may be imperfect. |
| Approach: | They propose a benchmark to measure the robustness of large language models under imperfect guidance. |
| Outcome: | The proposed benchmark compared LLM agents in navigation, cooking, and gaming in a variety of environments with auxiliary guidance and noisy or underspecified instructions extracted from demonstrations. |
LLM-based Open Domain Planning by Leveraging Entity-Attribute-Level Domain Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) based on Open domain Natural language planning have limited application potential. |
| Approach: | They propose a dataset with a baseline for Open domain Natural language planning . the dataset provides the largest dataset for textual procedures to date . |
| Outcome: | The proposed dataset provides the largest dataset for textual procedures to date . it leverages entity-attribute-level action models to reveal relevant physical properties . |
ADaPT: As-Needed Decomposition and Planning with Language Models (2024.findings-naacl)
Copied to clipboard
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, Tushar Khot
| Challenge: | Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. |
| Approach: | They propose an approach that explicitly plans and decomposes complex sub-tasks when the LLM is unable to execute them. |
| Outcome: | The proposed approach significantly outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |