Multimodal Embodied Plan Prediction Augmented with Synthetic Embodied Dialogue (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied task completion requires an agent to predict environment actions to complete tasks based on natural language instructions and egocentric visual observations. |
| Approach: | They propose a method to generate human-human dialogues and use them as training data for plan prediction. |
| Outcome: | The proposed model outperforms language-only models but falls short of oracle plans. |
Similar Papers
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion (2023.emnlp-main)
Copied to clipboard
Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, Alessandro Suglia
| Challenge: | Embodied MultiModal Agent (EMMA) is a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text generation. |
| Approach: | They propose an Embodied MultiModal Agent (EMMA) that uses a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text. |
| Outcome: | The proposed model performs on par with similar models on several VL benchmarks and sets a new state-of-the-art success rate on the Dialog-guided Task Completion (DTC) benchmark. |
Model-Based Imaginative Planning for Embodied Agents (2026.acl-long)
Copied to clipboard
Junru Song, Hengzhe Jin, Yucong Huang, Tingsong Jiang, Weien Zhou, Feifei Wang, Yang Yang, Ying Wen, Wen Yao
| Challenge: | a lightweight world model converts raw pixels into object-centric symbolic states amenable to language-based reasoning . IMPLEMENT is a framework for grounding language agents in visual embodied environments . |
| Approach: | They propose a model-based reasoning framework that enables frozen large language models to perform imaginative planning. |
| Outcome: | The proposed framework can be used to ground language agents in visual embodied environments. |
FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to embodied AI tend to learn policies from expert demonstrations, but without a mechanism to evaluate the quality of demonstrated actions, they are limited to learning from optimal behaviour or risk replicating errors and inefficiencies. |
| Approach: | They propose to embed language feedback into a Transformer-based policy and optionally complement the traditional next action prediction objective with auxiliary self-supervised learning objectives for feedback prediction. |
| Outcome: | The proposed method improves agents’ compositional generalisation abilities and robustness on a range of embodied Vision-and-Language tasks in a custom babyAI-XGen environment. |
LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. |
| Approach: | They propose a flexible and simulation-free testbed that simulates 6 representative embodied tasks in textual embodies. |
| Outcome: | The proposed testbed offers adaptability to diverse environments without multiple simulation engines and allows easy customization of communication and action strategies. |
A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for building multimodal open-domain dialogue agents based on large datasets are limited in real-world settings . |
| Approach: | They propose a new learning strategy called vision-language warm-up tasks for multimodal dialogue models that relies solely on learning from target data. |
| Outcome: | The proposed learning strategy achieves comparable and in some cases superior performance compared to existing state-of-the-art models on various evaluation metrics. |
Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | a paper presents a data-efficient solution to constructing task-oriented dialogue systems . large language models have shown success in modeling such dialogues, but they require large quantities of data . |
| Approach: | They propose a system that leverages explicit instructions from agent guidelines . they propose dialogue-document matching and action-oriented masked language modeling . |
| Outcome: | The proposed system improves accuracy predicting in- and out-of-distribution actions while preserving high performance in settings with low or sparse data. |
A Language-First Approach for Procedure Planning (2023.findings-acl)
Copied to clipboard
| Challenge: | Developing intelligent agents requires the ability to produce plans on the fly based on visual observations. |
| Approach: | They propose a language-first procedure planning framework with a modularized design . they first align current and goal observations with corresponding steps and then use a pre-trained LM to predict intermediate steps. |
| Outcome: | The proposed framework matches state-of-the-art procedures on COIN and CrossTask benchmarks. |
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on whether large language models are capable of planning or executing plans. |
| Approach: | They propose an abductive reasoning task using wikiHow to test the effectiveness of small models over large models. |
| Outcome: | The proposed task demonstrates the effectiveness of small models over large models in most scenarios. |
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained and frozen LLMs can effectively map simple scene re-arrangement instructions to programs over a robot’s visuomotor functions, but fixed prompts fall short. |
| Approach: | They propose an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting. |
| Outcome: | The proposed agent parses human-robot dialogue into action programs using retrieval-augmented LLM prompting. |
Multimodal Procedural Planning via Dual Text-Image Prompting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Embodied agents have demonstrated performance in following instructions informed by texts and images . however, the potential of models providing useful guidelines for humans to complete tasks remains underexplored . |
| Approach: | They propose a multimodal procedural planning task that generates paired text-image plans . this task provides more complementary and informative guidance than unimodal plans a . authors propose modality prompting methods that leverage zero-shot reasoning ability . |
| Outcome: | The proposed method improves the interaction in dual modalities and provides more information than unimodal plans. |