A Virtual Assistant for Architectural Design in a VR Environment (2026.eacl-demo)
Copied to clipboard
Ander Salaberria, Oier Ijurco, Markel Ferro, Jiayuan Wang, Iñigo Vilá Muñoz, Roberto de Ioris, Jeremy Barnes, Oier Lopez De Lacalle
| Challenge: | Existing interfaces lack intuitive, contextaware capabilities needed for easily navigating complex architectural components in 3D spaces. |
| Approach: | They propose a virtual assistant that allows users to interact with the BIM file in a VR environment. |
| Outcome: | The proposed assistant could allow users to interact with the BIM file in a virtual reality environment and perform complex actions. |
Similar Papers
CARD: Cross-modal Agent Framework for Generative and Editable Residential Design (2025.emnlp-main)
Copied to clipboard
| Challenge: | Architectural design automation has made significant progress, but the complexity of open-world environments makes residential design a challenging task. |
| Approach: | They propose a framework that leverages a system of specialized cross-modal agents to adapt to open-world residential design. |
| Outcome: | The proposed framework enables users to generate and edit residential design without requiring specialized expertise. |
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams (2023.acl-long)
Copied to clipboard
Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, Seungwhan Moon
| Challenge: | Existing models lack a large-scale benchmark to capture user–assistant interactions . et al., 2022: 145-160. |
| Approach: | They propose a video-grounded task-oriented dialog dataset that captures real-world AI-assisted user scenarios in VR. |
| Outcome: | The proposed dataset captures real-world AI-assisted user scenarios in VR. |
INDOORWORLD : Integrating Physical Task Solving and Social Simulation in A Heterogeneous Multi-Agent Environment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing virtual environments for LLM agent research focus on task solving or social simulation . existing environments for virtual environments lack physical grounding of social behaviors . |
| Approach: | They propose a virtual environment that tightly integrates physical and social dynamics . IndoorWorld is a heterogeneous multi-agent environment that integrates social and physical dynamics based on a simulation of physical environments . |
| Outcome: | The proposed environment integrates physical and social dynamics into a heterogeneous multi-agent environment. |
Open-Theatre: An Open-Source Toolkit for LLM-based Interactive Drama (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools for creating, modifying, and experimenting with interactive dramas are limited. |
| Approach: | They propose an open-source toolkit for creating configurable LLM-based interactive drama. |
| Outcome: | The proposed toolkit enhances narrative coherence and realistic behavior in interactions with agents. |
AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack immersion and adaptability, resulting in limited character orchestration and on-the-fly character introduction. |
| Approach: | They propose an LLM-based framework that allows actors to interact with users in an ongoing narrative. |
| Outcome: | The proposed framework outperforms commercial LLMs in character consistency, environment grounding, and narrative coherence. |
Connecting Language and Vision to Actions (P18-5)
Copied to clipboard
| Challenge: | Recent advances in language and vision have made incredible progress in describing images and interacting with visual content in a physical or embodied environment. |
| Approach: | This tutorial will provide an overview of the growing number of multimodal tasks and datasets that combine textual and visual understanding. |
| Outcome: | This tutorial will review the state-of-the-art approaches to selected tasks such as image captioning, visual question answering and visual dialog. |
LEGENT: Open Platform for Embodied Agents (2024.acl-demos)
Copied to clipboard
Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing integrations of large language models and large multimodal models are limited . Existing platforms for developing embodied agents are limited and limited based on open-source software. |
| Approach: | They propose an open platform for developing embodied agents using LLMs and LMMs. |
| Outcome: | The proposed platform surpasses GPT-4V in embodied tasks with its model training on LEGENT data. |
SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge Refinement (2025.acl-short)
Copied to clipboard
Runnan Fang, Xiaobin Wang, Yuan Liang, Shuofei Qiao, Jialong Wu, Zekun Xi, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen
| Challenge: | Using Large Language Models (LLMs)-based agents can enhance their understanding of environments and tasks. |
| Approach: | They propose a framework that allows agents to synthesize possible scenarios with multi-step action invocation within the action space and perform Monte Carlo Tree Search exploration to refine their action knowledge in the current environment. |
| Outcome: | The proposed framework synthesizes possible scenarios with multi-step action invocation within the action space and performs Monte Carlo Tree Search exploration to refine action knowledge in the current environment. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current task-oriented dialogue systems focus on multi-turn text/speech interaction, then call back-end APIs to perform task. |
| Approach: | They propose a GUI-based task-oriented dialogue system that can perform GUI operations on real APPs without invoking TOD-specific backend APIs. |
| Outcome: | The proposed GUI-based task-oriented dialogue system can perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. |