Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, Yoav Artzi
| Challenge: | a collaborative game with natural language instruction allows users to adapt to the system abilities by changing their language or deciding to accomplish tasks themselves. |
| Approach: | They propose a collaborative game where a user instructs a system to complete tasks, but acts alongside it. |
| Outcome: | The proposed game allows users to adapt to the system abilities by changing their language or deciding to accomplish tasks themselves. |
Similar Papers
Learning to execute instructions in a Minecraft dialogue (2020.acl-main)
Copied to clipboard
| Challenge: | Existing attempts to build interactive agents that can communicate with humans about and operate within the physical world are either completely ungrounded, focus on slot-value filling tasks, or operate within static environments, such as images or videos. |
| Approach: | They define the subtask of predicting correct action sequences in a given game context and capture B’s past actions as well as B’ s perspective leads to a significant improvement in performance. |
| Outcome: | The proposed task improves the performance of the two-player game and its corresponding Minecraft Dialogue Corpus. |
Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to map context-dependent sequential instructions to actions are based on discourse and state dependencies . we evaluate on SCONE domains and show absolute accuracy improvements of 9.8%-25.3% . |
| Approach: | They propose a model that considers previous utterances and the state of the world to map sequential instructions to actions. |
| Outcome: | The proposed model improves on the SCONE domains and on the target domains. |
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction (D18-1)
Copied to clipboard
| Challenge: | Existing models that map from inputs to actions are inefficient and require hand-crafted meaning representations. |
| Approach: | They propose to decompose instruction execution to goal prediction and action generation . they introduce two benchmarks for instruction following: LANI and CHAI . |
| Outcome: | The proposed model decomposes instruction execution to goal prediction and action generation. |
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In the Minecraft Collaborative Building Task, two players collaborate to build a building using 3D blocks. |
| Approach: | They propose to use large language models to model the Builder's sequence of actions in the Minecraft Collaborative Building Task. |
| Outcome: | The proposed methods significantly improve performance over baseline methods and provide detailed analysis for future work. |
Learning to Speak and Act in a Fantasy Text Adventure Game (D19-1)
Copied to clipboard
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston
| Challenge: | Existing studies on grounded dialogue use only statistical regularities of text data, without explicit understanding of the world that the text describes. |
| Approach: | They propose a large-scale crowdsourced text adventure game as a research platform for studying grounded dialogue. |
| Outcome: | The proposed game allows agents to perceive, emote, and act whilst conducting dialogue with other agents. |
Task-aware Retrieval with Instructions (2023.findings-acl)
Copied to clipboard
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih
| Challenge: | Existing models that learn intents from labeled data are complicated and require a vast number of annotated examples to train a model. |
| Approach: | They propose a general-purpose task-aware retrieval system with instructions that can adapt to a new task without any parameter updates. |
| Outcome: | The proposed system outperforms two benchmarks on a set of domains and tasks on X2-Retrieval. |
Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior (2021.tacl-1)
Copied to clipboard
| Challenge: | Despite its potential and prevalence, this signal is understudied for learning to generate natural language. |
| Approach: | They propose to use this signal to improve the system's ability to generate instructions via contextual bandit learning. |
| Outcome: | The proposed system improves its ability to generate natural language through interaction with users, and the results are shown. |
Deep Dungeons and Dragons: Learning Character-Action Interactions from Role-Playing Game Transcripts (N18-2)
Copied to clipboard
| Challenge: | a novel approach to understanding narratives involves modelling the interaction between characters and actions . we propose role-playing games as a testbed for inferring interactions between characters in narratives . |
| Approach: | They propose role-playing games as a testbed for learning latent ties between characters and actions . they propose to combine character and action descriptions from online discussion forums . |
| Outcome: | The proposed model can capture interactions between characters and actions in narratives . it can predict actions better when character attributes are taken into account . |
Learning Action Conditions from Instructional Manuals for Instruction Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | a weakly supervised task is proposed to extract mentions of preconditions and postconditions of actions from instructional manuals. |
| Approach: | They propose a task dubbed action condition inference which extracts mentions of preconditions and postconditions of actions from instructional manuals. |
| Outcome: | The proposed approach improves on the existing models, but still far behind human performance. |
Sharing the Cost of Success: A Game for Evaluating and Learning Collaborative Multi-Agent Instruction Giving and Following Policies (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have led to language model-based systems that do a good job at creating natural dialogue behaviour but are often verbose and brittle. |
| Approach: | They propose a game that requires two players to coordinate on vision and language observations. |
| Outcome: | The proposed game achieves high success rates when bootstrapped with heuristic partner behaviors that implement insights from the analysis of human-human interactions. |