OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following (2024.acl-long)
Copied to clipboard
| Challenge: | Embodied Instruction Following (EIF) is a crucial task in embodied learning . however, there is n'a unified understanding regarding the impact of various components on task performance . |
| Approach: | They propose a framework that delineates the core components essential for embodied learning tasks . they integrate a multi-agent design into the Planner component of their LLM-centric architecture . |
| Outcome: | OPEx delineates the core components essential for solving embodied learning tasks . integrating a multi-agent design into the Planner component of the LLM-centric architecture further elevates performance. |
Similar Papers
DANLI: Deliberative Agent for Following Natural Language Instructions (2022.emnlp-main)
Copied to clipboard
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, Joyce Chai
| Challenge: | Recent work on embodied AI agents that can perform tasks by following human language instructions is limited by reactive methods, which are insufficient for long-horizon complex tasks. |
| Approach: | They propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience. |
| Outcome: | The proposed agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark. |
ALFRED-L: Investigating the Role of Language for Action Learning in Interactive Visual Environments (2022.emnlp-main)
Copied to clipboard
Arjun Akula, Spandana Gella, Aishwarya Padmakumar, Mahdi Namazifar, Mohit Bansal, Jesse Thomason, Dilek Hakkani-Tur
| Challenge: | Embodied Vision and Language Task Completion requires an embodied agent to interpret natural language instructions and egocentric visual observations to navigate through and interact with environments. |
| Approach: | They propose to use ALFRED to test whether models can adapt to tasks not seen during training that require the same types of language understanding as ALFred. |
| Outcome: | The proposed model can adapt to tasks that require the same types of language understanding as ALFRED-L. |
Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experiments with 4 different LLMs across 5 embodied environments show significant efficiency improvements, with only minor drops in agent performance. |
| Approach: | They propose an intrinsic method that injects exit instructions during generation and an extransic system that verifies task completion to determine when to halt an agent’s trial. |
| Outcome: | The proposed method injects exit instructions during generation and an exit method verifies task completion to determine when to halt an agent’s trial. |
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized natural language processing with impressive capabilities, but they lack domain specificity, real-time information and face challenges in solving specialized problems. |
| Approach: | They propose a multi-LLM approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. |
| Outcome: | The proposed model outperforms existing models by demonstrating its effectiveness and advantages in tool learning. |
A Review of Prominent Paradigms for LLM-Based Agents: Tool Use, Planning (Including RAG), and Feedback Learning (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models have been used for planning, tool use, and feedback learning . inconsistent taxonomy and complexity of workflows create challenges . |
| Approach: | They propose a unified taxonomy to review and discuss the three paradigms . they define environments/tasks, common LLM-profiled roles and universally applicable workflows based on prior work . |
| Outcome: | The proposed taxonomy compares LMPR implementations and workflow usage across paradigms . large language models have human-like reasoning capabilities, the authors say . |
Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for grounding pretrained language models as task planners are challenging due to their intricate entanglement with domain knowledge. |
| Approach: | They propose a framework that leverages the hierarchical nature of semantic skills to ground them in different domains. |
| Outcome: | The proposed framework is effective in 300 cross-domain EIF scenarios. |
xLAM: A Family of Large Action Models to Empower AI Agent Systems (2025.naacl-long)
Copied to clipboard
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Quoc Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R N, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Autonomous agents powered by large language models (LLMs) have attracted significant research interest, but there are few standards for developing specialized models for agent tasks. |
| Approach: | They propose a series of large action models with dense and mixture-of-expert architectures that unifies, augments, and synthesizes diverse datasets to enhance agent generalizability and performance. |
| Outcome: | The proposed models outperform GPT-4, Claude-3, and many other models in terms of tool use and outperformed GPT-based models on multiple agent ability benchmarks. |
LangSuit·E: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. |
| Approach: | They propose a flexible and simulation-free testbed that simulates 6 representative embodied tasks in textual embodies. |
| Outcome: | The proposed testbed offers adaptability to diverse environments without multiple simulation engines and allows easy customization of communication and action strategies. |
Learning to Use Tools via Cooperative and Interactive Agents (2024.findings-emnlp)
Copied to clipboard
Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Pengjie Ren, Suzan Verberne, Zhaochun Ren
| Challenge: | Existing methods for large language models (LLMs) use one agent to iterate and execute tools, but they suffer from performance degradation when addressing practical tasks. |
| Approach: | They propose a tool learning framework that coordinates three specialized agents for tool selection, tool execution, and action calibration separately. |
| Outcome: | The proposed framework outperforms baseline models on three datasets with 14% higher success rate. |
LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following (2023.emnlp-main)
Copied to clipboard
| Challenge: | Embodied Instruction Following has shown an impressive success rate when the environment has been seen in training, but when deployed in an unseen environment, it tends to struggle when deployed with an unsightly environment. |
| Approach: | They propose to explicitly align the agent’s hidden states with the instructions via contrastive learning to bridge the semantic gap between high-level language instructions and the agent's low-level action space. |
| Outcome: | The proposed meta-actions achieve a 4.5% success rate in unseen environments compared to a strong multi-modal Transformer baseline . |