Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods struggle to ensure semantic correctness and rely on simple or unrealistic datasets. |
| Approach: | They propose a benchmark to evaluate language models’ ability to generate PDDL code from natural language descriptions of planning tasks. |
| Outcome: | The proposed benchmark evaluates the ability of language models to generate PDDL code from natural language descriptions of planning tasks against ground truth and a dataset of 145,918 text-to-PDDL pairs with varying levels of difficulty. |
Similar Papers
On the Limit of Language Models as Planning Formalizers (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models can create plans that are neither executable nor verifiable in grounded environments. |
| Approach: | They use Large Language Models to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). |
| Outcome: | The proposed model outperforms the models directly generating plans while being robust to lexical perturbation. |
PlanningArena: A Modular Benchmark for Multidimensional Evaluation of Planning and Tool Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can be significantly improved by integrating external tools. |
| Approach: | They propose a framework that integrates external tools into large language models to evaluate their ability to generate action plans. |
| Outcome: | The proposed framework evaluates the ability of large language models to generate action plans and generate action plan templates. |
Language Model as Planner and Formalizer under Constraints (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely used in planning but lack interpretability and control. |
| Approach: | They propose to augment widely used planning benchmarks with manually annotated, fine-grained, and rich natural language constraints spanning four formally defined categories. |
| Outcome: | The proposed model outperforms existing models in 4 state-of-the-art reasoning LLMs, 4 formal languages, and 4 datasets. |
Can LLMs Truly Plan? A Comprehensive Evaluation of Planning Capabilities (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing assessments of planning capabilities of large language models are limited to single-language or specific representation formats. |
| Approach: | a new benchmark is developed to assess the planning capabilities of large language models. |
| Outcome: | The Multi-Plan benchmark highlights performance disparities among models . language differences showed minimal impact, while mathematically structured representations improved accuracy . |
Iterative Formalization and Planning in Partially Observable Environments (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to formalize an environment into the Planning Domain Definition Language (PDDL) have been shown to improve performance and control. |
| Approach: | They propose a framework to iteratively formalize, plan, grow, and refine PDDL representations by decomposing the environment and the goal into fully observable episodes. |
| Outcome: | The proposed framework improves planning success and exhibits robustness against problem complexity compared to end-to-end approaches. |
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on whether large language models are capable of planning or executing plans. |
| Approach: | They propose an abductive reasoning task using wikiHow to test the effectiveness of small models over large models. |
| Outcome: | The proposed task demonstrates the effectiveness of small models over large models in most scenarios. |
PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing Vision-Language Models (VLMs) fail to analyze planning maps . specialized visual representations of land use zones, transportation networks, and development policies are needed to interpret complex planning maps. |
| Approach: | They propose a domain-specific VLM tailored for urban planning maps that employs three innovations: PlanAnno-V framework for high-quality VQA data synthesis, Critical Point Thinking (CPT) and PlanBench-V benchmark for systematic evaluation. |
| Outcome: | The new model outperforms general-purpose VLMs on planning map interpretation tasks. |
Unifying Inference-Time Planning Language Generation (2026.findings-acl)
Copied to clipboard
Prabhu Prakash Kagitha, Bo Sun, Ishan Desai, Andrew Zhu, Cassie Huang, Manling Li, Ziyang Li, Li Zhang
| Challenge: | Large language models (LLMs) are used to generate a formal representation of a plan in a planning language. |
| Approach: | They propose a unifying organizational framework based on intermediate representations to unify the inference-time LLM-as-formalizer methodology for classical planning. |
| Outcome: | The proposed framework subsumes most existing work and proposes new ones that involve syntactically similar but high-resource intermediate languages. |
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning. |
| Approach: | They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions. |
| Outcome: | The proposed frameworks are used to support reliable off-the-shelf AP planners. |
GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated significant capabilities in processing and understanding text data. |
| Approach: | They propose a structure-based instruction-based method to enhance LLM performance on complex graph tasks. |
| Outcome: | The proposed framework outperforms open-source models on graph problem-solving, but the gap is narrowing. |