Text2World: Benchmarking Large Language Models for Symbolic World Model Generation (2025.findings-acl)
Copied to clipboard
Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Yao Mu, Hongyuan Zhang, Wenqi Shao, Ping Luo
| Challenge: | Recent studies have encountered limitations in leveraging large language models to generate symbolic world models. |
| Approach: | They propose a benchmarking framework based on planning domain definition language (PDDL) that employs multi-criteria, execution-based metrics for a more robust evaluation. |
| Outcome: | The proposed model outperforms models trained with large-scale reinforcement learning, but lacks the robustness needed to perform in world modeling. |
Similar Papers
Can Language Models Serve as Text-Based World Simulators? (2024.acl-short)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have pointed towards an alternative approach by leveraging the huge amount of knowledge contained in their pre-training datasets. |
| Approach: | They build and use a benchmark to quantify how well text-based simulators can serve as text-driven world simulators. |
| Outcome: | The proposed benchmark aims to quantify how well language models can serve as world simulators. |
From Word to World: Can Large Language Models be Implicit Text-based World Models? (2026.acl-long)
Copied to clipboard
Yixia Li, Hongru Wang, Jiahao Qiu, Zhenfei Yin, Dongdong Zhang, Cheng Qian, Zeping Li, Xiaoteng Ma, Guanhua Chen, Heng Ji
| Challenge: | Agentic learning increasingly hinges on interaction, yet real-world experience is expensive, limited, and often irreversible at inference time. |
| Approach: | They propose a framework that reframes language modeling as next-state prediction under interaction. |
| Outcome: | The proposed framework evaluates world models in text-based environments . it shows that sufficiently trained models capture coherent environment dynamics . |
Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey (2024.acl-long)
Copied to clipboard
Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, 林一冠 林一冠, Bin Xu, Ren Bowen, Chong Feng, Yang Gao, Heyan Huang
| Challenge: | Existing benchmark-based evaluations cannot accurately reflect the performance of real-world applications. |
| Approach: | They propose a reliable strategy for domains to choose more robust LLMs for real-world applications. |
| Outcome: | The proposed strategy addresses the challenges faced by domains to choose more robust LLMs for real-world applications. |
On the Limit of Language Models as Planning Formalizers (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models can create plans that are neither executable nor verifiable in grounded environments. |
| Approach: | They use Large Language Models to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). |
| Outcome: | The proposed model outperforms the models directly generating plans while being robust to lexical perturbation. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language. |
| Approach: | They propose a model that integrates symbolic data into LLM training without loss of generality ability. |
| Outcome: | The proposed model performs better on symbol- and NL-centric tasks. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Making Large Language Models into World Models with Precondition and Effect Knowledge (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are not inherently designed to model real-world dynamics, but can be induced to perform two critical world model functions: determining the applicability of an action based on a given world state and predicting the resulting world state upon action execution. |
| Approach: | They propose to use Large Language Models to model world states and preconditions . they validate that precondition and effect knowledge generated by LLMs aligns with human understanding of world dynamics . |
| Outcome: | The proposed model can predict valid actions and state transitions, thereby replicating existing models. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |