| Challenge: | Using computer games to simulate grounded situations, we want to develop interactive agents that can communicate with humans to solve tasks in grounded scenarios. |
| Approach: | They propose a Minecraft-based collaborative building task in which one player is shown a building structure and needs to instruct the other player to build it. |
| Outcome: | The proposed agent can communicate with humans to solve a building task in a Minecraft-based environment without the need for physical robots. |
Similar Papers
Learning to execute instructions in a Minecraft dialogue (2020.acl-main)
Copied to clipboard
| Challenge: | Existing attempts to build interactive agents that can communicate with humans about and operate within the physical world are either completely ungrounded, focus on slot-value filling tasks, or operate within static environments, such as images or videos. |
| Approach: | They define the subtask of predicting correct action sequences in a given game context and capture B’s past actions as well as B’ s perspective leads to a significant improvement in performance. |
| Outcome: | The proposed task improves the performance of the two-player game and its corresponding Minecraft Dialogue Corpus. |
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Creating embodied, situated agents able to move in, communicate naturally about, and collaborate on human terms in the physical world has been a persisting goal in artificial intelligence (Winograd, 1972). |
| Approach: | They propose to use a 3D Minecraft dataset to model the beliefs of human partners in situ to enable theory of mind modeling in situated interactions. |
| Outcome: | The proposed model can be used to model human collaborative behaviors in the 3D virtual blocks world of Minecraft. |
Learning to Execute Actions or Ask Clarification Questions (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing work on Minecraft Corpus Dataset only learns to execute instructions neglecting the importance of asking for clarifications. |
| Approach: | They propose to annotate all builder utterances into eight types, including clarification questions, and propose a builder agent model capable of determining when to ask or execute instructions. |
| Outcome: | The proposed model outperforms existing models on the collaborative building task with a substantial improvement. |
Discourse Structure for the Minecraft Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | a discourse annotated version of the Minecraft Dialogue Corpus is a new linguistic resource for human-computer interaction . a recent study shows that inferring excecutable actions from language is difficult in the Minecraft setting . |
| Approach: | They propose a discourse annotated version of the Minecraft Dialogue Corpus . they train a parser with a novel "2 pass architecture" that gives excellent results . |
| Outcome: | The proposed model performs well on attachment prediction and relation labeling tasks especially long distance attachments. |
Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback (2024.findings-eacl)
Copied to clipboard
| Challenge: | In many approaches to Natural Language Processing tasks, language is inherently interactive. |
| Approach: | They propose to use human-AI collaboration to improve human-human interaction by providing feedback that the agent can understand and utilize. |
| Outcome: | The proposed task is an interactive grounded language understanding task in a MineCraft-like world. |
Dialogue is the Plan: From Interface to Joint Action in Agentic AI (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Model agents' language use is often used as an interface for instructing and reporting results. |
| Approach: | They argue that large language models are often used as an interface for instructingactions and reporting results. |
| Outcome: | We show that large-scale language models can be used to plan and act, yet their language is often used as an interface for instructing and reporting results. |
Dialogue Collection for Recording the Process of Building Common Ground in a Collaborative Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on the process of building common ground have not been well conducted. |
| Approach: | They propose a method for recording the process of building common ground through a dialogue by using the intermediate result of a task. |
| Outcome: | The proposed method can record the building common ground process by using the intermediate result of a task and can be estimated quite accurately. |
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In the Minecraft Collaborative Building Task, two players collaborate to build a building using 3D blocks. |
| Approach: | They propose to use large language models to model the Builder's sequence of actions in the Minecraft Collaborative Building Task. |
| Outcome: | The proposed methods significantly improve performance over baseline methods and provide detailed analysis for future work. |
Achieving Common Ground in Multi-modal Dialogue (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial focuses on three main topic areas: grounding in human-human communication, dialogue systems and multi-modal interactive systems. |
| Approach: | This tutorial examines the use of computational dialogue research to design grounding modules and behaviors in cutting-edge systems. |
| Outcome: | This tutorial examines the results of recent research on grounding in human-human communication . it shows how these results lead to rich and challenging opportunities for doing grounding more flexible and powerful ways . |
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2020.acl-main)
Copied to clipboard
| Challenge: | a set of 12 tasks that measure if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, and perceive and converse about images. |
| Approach: | They propose a set of 12 tasks that measure if a conversational agent can communicate engagingly with personality and empathy . they use large dialogue datasets to multi-task and obtain state-of-the-art results . |
| Outcome: | The proposed model improves over a BERT pre-trained model on large dialogue datasets and provides state-of-the-art results on many of the tasks. |