UnrealLLM: Towards Highly Controllable and Interactable 3D Scene Generation by LLM-powered Procedural Content Generation

Challenge:	UnrealLLM is a novel framework that connects natural language descriptions with the professional PCG system (Unreal Engine 5) to automate scene generation.
Approach:	They propose a novel multi-agent framework that connects natural language descriptions with the professional PCG system (Unreal Engine 5) to automate scene generation.
Outcome:	The proposed framework achieves competitive performance in technical metrics and aesthetic quality, offering unique advantages in generation scale and interactivity.

Similar Papers

Overview

Challenge

Approach

Outcome

AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems (2026.findings-acl)

Copied to clipboard

Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, Gangyi Ding

Information Retrieval

Challenge:	Recent advances in Large Language Models (LLMs) and generative models have motivated studies on automated game generation from natural language descriptions.
Approach:	They propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation.
Outcome:	The proposed system covers model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation.

SceneLM: 3D-Aware Language Models for Editable 3D Scene Synthesis (2026.findings-acl)

Copied to clipboard

Xingbo Yao, Xiaoyu Chen, Doudou Zhang, Mingzhi Sheng, Boyuan Cao, Ying-Cong Chen, Hui Xiong

Multimodality

Challenge:	Existing methods for synthesising 3D scenes from a single image are text-driven and lack precise metric understanding from images.
Approach:	They propose a language-model-based framework that grounds 3D scene synthesis in visual evidence by recovering an executable metric 3D layout directly from a single image.
Outcome:	The proposed framework recovers an executable metric 3D layout directly from an RGB image and instantiates, places, and edits objects for iterative refinement.

Open-Theatre: An Open-Source Toolkit for LLM-based Interactive Drama (2025.emnlp-demos)

Copied to clipboard

Tianyang Xu, Hongqiu Wu, Weiqi Wu, Hai Zhao

Dialogue Systems & Conversational Agents

Challenge:	Existing tools for creating, modifying, and experimenting with interactive dramas are limited.
Approach:	They propose an open-source toolkit for creating configurable LLM-based interactive drama.
Outcome:	The proposed toolkit enhances narrative coherence and realistic behavior in interactions with agents.

LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study (2025.acl-long)

Copied to clipboard

Dongil Yang, Minjin Kim, Sunghwan Kim, Beong-woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, Jinyoung Yeo

Multimodality

Challenge:	Large language models (LLMs) have demonstrated impressive progress in various text-based tasks, such as question-answering and content generation.
Approach:	They propose a benchmark to evaluate Large Language Models’ ability to understand scene graphs and generate them from textual narratives.
Outcome:	The proposed model performs well on scene graph understanding but struggles with scene graph generation, particularly for complex narratives.

Towards Enhanced Immersion and Agency for LLM-based Interactive Drama (2025.acl-long)

Copied to clipboard

Hongqiu Wu, Weiqi Wu, Tianyang Xu, Jiameng Zhang, Hai Zhao

Natural Language Interfaces

Challenge:	Existing studies have focused on the role of immersion and agency in interactive drama.
Approach:	They propose a playwriting-guided generation method that helps LLMs craft dramatic stories with substantially improved structures and narrative quality.
Outcome:	The proposed method improves storytelling quality and immersion and agency, while allowing agents to refine their reactions to align with the player’s intentions.

SceneGenAgent: Precise Industrial Scene Generation with Coding Agent (2025.acl-long)

Copied to clipboard

Xiao Xia, Dan Zhang, Zibo Liao, Zhenyu Hou, Tianrui Sun, Jing Li, Ling Fu, Yuxiao Dong

Language Models

Challenge:	Recent work on scene generation focuses on generating 3D scenes from textual descriptions . however, the task of generating industrial scenes with LLMs is complex and requires precise measurements and positioning .
Approach:	They propose an LLM-based agent for generating industrial scenes through C# code.
Outcome:	Experiments show that LLMs powered by SceneGenAgent exceed their original performance . the agent achieves 81.0% success rate in real-world industrial scene generation tasks .

AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing (2026.findings-acl)

Copied to clipboard

Zhenhua Xu, Dongsheng Chen, Shuo Wang, Jian Li, Chengjie Wang, Meng Han, Yabiao Wang

Challenge:	Existing LLMs lack immersion and adaptability, resulting in limited character orchestration and on-the-fly character introduction.
Approach:	They propose an LLM-based framework that allows actors to interact with users in an ongoing narrative.
Outcome:	The proposed framework outperforms commercial LLMs in character consistency, environment grounding, and narrative coherence.

UCTG: A Unified Controllable Text Generation Framework for Query Auto-Completion (2025.coling-industry)

Copied to clipboard

Zhipeng Li, Shuang Zheng, Jiaping Xiao, Xianneng Li, Lei Wang

Low-Resource NLP

Challenge:	Existing approaches to control text generation (CTG) are essentially challenging to adapt to various control objectives and constraints, which results in mixed success.
Approach:	They propose a unified controllable text generation framework which integrates a control module, a prompt module, and a generation module.
Outcome:	The proposed framework significantly improves query accuracy and coherence in tasks with different objectives and constraints.

MapCoder: Multi-Agent Code Generation for Competitive Problem Solving (2024.acl-long)

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

Programming Languages in NLP

Challenge:	Large language models (LLMs) have impressive proficiency in natural language processing, but performance in code generation tasks remains limited.
Approach:	They propose a framework that emulates the full cycle of program synthesis as observed in humans.
Outcome:	The proposed framework replicates the full cycle of program synthesis as observed in human developers.

AIPOM: Agent-aware Interactive Planning for Multi-Agent Systems (2025.emnlp-demos)

Copied to clipboard

Hannah Kim, Kushan Mitra, Chen Shen, Dan Zhang, Estevam Hruschka

Challenge:	Large language models (LLMs) are being used for planning in orchestrated multi-agent systems . existing LLMs fall short of human expectations and lack effective mechanisms for users to inspect, understand, and control their behaviors.
Approach:	They propose a system supporting human-in-the-loop planning through conversational and graph-based interfaces.
Outcome:	AIPOM enables users to transparently inspect, refine, and collaboratively guide LLM-generated plans, significantly enhancing user control and trust in multi-agent workflows.