ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for embodied agents focus on directly executing instructions without considering whether objects can be manipulated. |
| Approach: | They propose a benchmark that evaluates embodied agents in dynamic environments . they use plug-and-play module that augments existing planners with explicit affordance reasoning . |
| Outcome: | The proposed benchmark evaluates embodied agents in dynamic environments with unpredictable affordances . ADAPT significantly improves robustness and task success across seen and unseen environments . |
Similar Papers
Open-World Planning via Lifted Regression with LLM-Inferred Affordances for Embodied Agents (2025.acl-long)
Copied to clipboard
Xiaotian Liu, Ali Pesaranghader, Hanze Li, Punyaphat Sukcharoenchaikul, Jaehong Kim, Tanmana Sadhu, Hyejeong Jeon, Scott Sanner
| Challenge: | Existing open-world planning methods rely on closed-world assumption (CWA) symbolic planners face combinatorial explosion of states and actions due to reliance on grounding. |
| Approach: | They propose an open-world planning approach integrating lifted regression with LLM-generated affordances. |
| Outcome: | The proposed approach outperforms state-of-the-art LLM planners and a grounded planner on three benchmarks. |
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of Large Language Models (LLMs) focus on task completion, but neglect a crucial capability: the ability to devise and adjust cost-optimal plans in response to changing environments. |
| Approach: | They propose a scalable, cost-centric benchmark to evaluate agents’ economic reasoning and replanning abilities. |
| Outcome: | Evaluating leading open-sourced and proprietary models on CostBench reveals a substantial gap in cost-aware planning . |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
Model-Based Imaginative Planning for Embodied Agents (2026.acl-long)
Copied to clipboard
Junru Song, Hengzhe Jin, Yucong Huang, Tingsong Jiang, Weien Zhou, Feifei Wang, Yang Yang, Ying Wen, Wen Yao
| Challenge: | a lightweight world model converts raw pixels into object-centric symbolic states amenable to language-based reasoning . IMPLEMENT is a framework for grounding language agents in visual embodied environments . |
| Approach: | They propose a model-based reasoning framework that enables frozen large language models to perform imaginative planning. |
| Outcome: | The proposed framework can be used to ground language agents in visual embodied environments. |
DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints (2026.acl-long)
Copied to clipboard
Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, ChenXu Lv, Junyang Lin
| Challenge: | Existing LLM planning benchmarks emphasize local, step-level reasoning rather than global constrained optimization. |
| Approach: | They propose a benchmark for practical long-horizon agent planning that uses local constrained reasoning and global constrained optimization. |
| Outcome: | The proposed benchmarks show that even frontier agentic LLMs struggle with these problems. |
VestaBench: An Embodied Benchmark for Safe Long-Horizon Planning Under Multi-Constraint and Adversarial Settings (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing safety benchmarks do not represent a diverse range of multi-constraint tasks that require long-horizon planning with a focus on safety. |
| Approach: | They propose a benchmark to assess the safety of embodied AI agents under multiple constraints. |
| Outcome: | The proposed benchmarks show that LLMs perform poorly against their tasks . they also suffer significantly compromised safety outcomes . |
Communication-Efficient Desire Alignment for Proactive Embodied Human–Agent Interaction (2026.acl-long)
Copied to clipboard
| Challenge: | Effective real-world human–agent interactions are long-term and repeated. |
| Approach: | They propose a simulation that uses a proxy user with value-driven preferences and natural language behavior to evaluate how agents adapt to users across interactions and satisfy their desires. |
| Outcome: | HA-Desire, a home assistance simulation, shows that agents can adapt to user needs and provide proactive assistance within limited communication. |
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments (2022.coling-1)
Copied to clipboard
| Challenge: | embodied AI tasks require a strong understanding of verbs and their corresponding actions. |
| Approach: | They propose a controlled benchmark for embodied action understanding using a simulated environment and a visual feature extractor. |
| Outcome: | The proposed benchmark achieves 81.4% accuracy and high inter-annotator agreement . the proposed model falls behind human models in a zero-shot scenario . |
ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for embodied spatial reasoning and long-term planning are non-trivial due to the combinatorial complexity of long-horizon abstract reasoning. |
| Approach: | They propose a large-scale benchmark for partially observable embodied spatial reasoning and long-term planning with large language models and vision language models. |
| Outcome: | The proposed model performs better in 16 task types, 5,000 rooms, and over 10 million evaluation trajectories with diverse data distribution. |
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)
Copied to clipboard
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang
| Challenge: | Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics. |
| Approach: | They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks. |
| Outcome: | The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring. |