Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback (2024.findings-eacl)
Copied to clipboard
| Challenge: | In many approaches to Natural Language Processing tasks, language is inherently interactive. |
| Approach: | They propose to use human-AI collaboration to improve human-human interaction by providing feedback that the agent can understand and utilize. |
| Outcome: | The proposed task is an interactive grounded language understanding task in a MineCraft-like world. |
Similar Papers
Improving Natural Language Interaction with Robots Using Advice (N19-1)
Copied to clipboard
| Challenge: | Recent studies focus on learning models for physically grounded language understanding tasks such as the blocks world domain. |
| Approach: | They propose a protocol for including advice, high-level observations about the task, which can help constrain the agent’s prediction. |
| Outcome: | The proposed approach can be extended to include advice, high-level observations about the task, and reduce the effort involved in supplying the advice. |
MM-IGLU: Multi-Modal Interactive Grounded Language Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | In human-robot interaction, a robot interprets user commands related to its environment, aiming to discern whether a specific command can be executed. |
| Approach: | They propose to integrate user statements with environment's description to create a multi-modal interactive Grounded language understanding model that integrates both visual and textual data. |
| Outcome: | The proposed model integrates user’s statement with environment’s description and a cutting-edge Multi-Modal Large Language Model merges both visual and textual data. |
tagE: Enabling an Embodied Agent to Understand Human Instructions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for natural language understanding (NLU) are limited due to the inherent ambiguity and incompleteness inherent in natural language. |
| Approach: | They propose a system to extract tasks from natural language instructions and map them to robots' established collection of skills. |
| Outcome: | The proposed system outperforms baseline models in the training and evaluation of a dataset featuring complex instructions. |
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches (2023.findings-emnlp)
Copied to clipboard
| Challenge: | People rely heavily on context to enrich meaning beyond what is literally said. |
| Approach: | They analyze how task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. |
| Outcome: | The proposed frameworks are based on linguistic goals, environmental contexts, and communicative affordances to enrich linguistic meaning. |
Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on how large language model agents collaborate with humans in equal roles emphasize the importance of coordination and communication. |
| Approach: | They propose to use chain-of-thought prompts to evaluate different collaboration perspectives, from independent to more complex, dependent tasks. |
| Outcome: | The proposed model significantly improves the evaluation metric. |
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021.naacl-main)
Copied to clipboard
| Challenge: | Empirical results show that today’s language models struggle at TuringAdvice . language models are getting ever-larger, and are being trained on ever-increasing quantities of text . |
| Approach: | They propose a task task that requires models to generate helpful advice in natural language. |
| Outcome: | The proposed model outperforms even multibillion parameter models on 600k in-domain training examples. |
Collaborative Dialogue in Minecraft (P19-1)
Copied to clipboard
| Challenge: | Using computer games to simulate grounded situations, we want to develop interactive agents that can communicate with humans to solve tasks in grounded scenarios. |
| Approach: | They propose a Minecraft-based collaborative building task in which one player is shown a building structure and needs to instruct the other player to build it. |
| Outcome: | The proposed agent can communicate with humans to solve a building task in a Minecraft-based environment without the need for physical robots. |
Grounding Language in Multi-Perspective Referential Communication (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a dataset of 2,970 human-written referring expressions, we find that the performance of automated models in both reference generation and comprehension lags behind that of pairs of human agents. |
| Approach: | They propose a task and dataset for referring expression generation and comprehension in multi-agent embodied environments where two agents must take into account one another's visual perspective to produce and understand references to objects in a scene. |
| Outcome: | The proposed model outperforms the strongest proprietary model and improves communicative success from 58.9 to 69.3% when trained with a listener. |
A Scalable Framework for Learning From Implicit User Feedback to Improve Natural Language Understanding in Large-Scale Conversational AI Systems (2021.emnlp-main)
Copied to clipboard
Sunghyun Park, Han Li, Ameen Patel, Sidharth Mudgal, Sungjin Lee, Young-Bum Kim, Spyros Matsoukas, Ruhi Sarikaya
| Challenge: | Existing methods to improve NLU are laborintensive and expensive. |
| Approach: | They propose a scalable and automatic approach to improving NLU in a large-scale conversational AI system by leveraging implicit user feedback. |
| Outcome: | The proposed framework improves NLU in a large-scale conversational AI system across 10 domains. |
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Creating embodied, situated agents able to move in, communicate naturally about, and collaborate on human terms in the physical world has been a persisting goal in artificial intelligence (Winograd, 1972). |
| Approach: | They propose to use a 3D Minecraft dataset to model the beliefs of human partners in situ to enable theory of mind modeling in situated interactions. |
| Outcome: | The proposed model can be used to model human collaborative behaviors in the 3D virtual blocks world of Minecraft. |