META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current task-oriented dialogue systems focus on multi-turn text/speech interaction, then call back-end APIs to perform task. |
| Approach: | They propose a GUI-based task-oriented dialogue system that can perform GUI operations on real APPs without invoking TOD-specific backend APIs. |
| Outcome: | The proposed GUI-based task-oriented dialogue system can perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. |
Similar Papers
Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Conversational assistants are increasingly popular across diverse real-world applications . speech data constitute high-dimensional signals that are difficult to model even for frontier models . |
| Approach: | They propose a data-centric customization approach for enhancing multimodal understanding in conversational speech modeling. |
| Outcome: | The proposed model achieves state-of-the-art on the Spoken-SQuAD benchmark using 10% of training data with open-weight models. |
Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems are drawing more attention in recent studies . current evaluation methods use annotated utterances in multi-turn dialogue sessions . |
| Approach: | They propose an interactive evaluation framework for task-oriented dialogue (TOD) they build a goal-oriented user simulator based on pre-trained models and use it to generate dialogues . |
| Outcome: | The proposed evaluation framework achieves 98% inform and success rates in the multi-woz dataset and the proposed scores measure the response quality besides the inform and successful rates. |
TOD-Flow: Modeling the Structure of Task-Oriented Dialogues (2023.emnlp-main)
Copied to clipboard
Sungryull Sohn, Yiwei Lyu, Anthony Liu, Lajanugen Logeswaran, Dong-Ki Kim, Dongsub Shim, Honglak Lee
| Challenge: | Recent advances in task-oriented dialogue systems have limitations regarding transparency and controllability. |
| Approach: | They propose to infer the TOD-flow graph from dialog data annotated with dialog acts and integrate it with any dialogue model to improve its prediction performance, transparency, and controllability. |
| Outcome: | The proposed approach improves dialog act classification and response generation performance in the MultiWOZ and SGD benchmarks. |
I know you are different! Towards Persona Driven Knowledge-infused Dialogue Assistant (2026.eacl-long)
Copied to clipboard
| Challenge: | Task-Oriented Dialogue (TOD) systems often fall short in delivering personalized, context-rich responses, especially in low-resource, code-mixed, and multimodal settings like Hinglish. |
| Approach: | They propose a Hinglish multimodal, multidomain, persona-based TOD dataset that captures user-agent interactions across text and visual modalities. |
| Outcome: | The proposed framework outperforms standard and ablated models in Hinglish and Hinglanish. |
ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format (2023.emnlp-demo)
Copied to clipboard
Qi Zhu, Christian Geishauser, Hsien-chin Lin, Carel van Niekerk, Baolin Peng, Zheng Zhang, Shutong Feng, Michael Heck, Nurul Lubis, Dazhen Wan, Xiaochen Zhu, Jianfeng Gao, Milica Gasic, Minlie Huang
| Challenge: | Existing tools for building TOD systems often lack a user-friendly interface . a toolkit with advanced, easily integrable modules is needed to bridge this gap . |
| Approach: | They propose a multifaceted dialogue system toolkit that integrates diverse datasets and models with a streamlined training process and in-depth evaluation tools. |
| Outcome: | The proposed toolkit combines RL and transfer learning to support the rapid development and evaluation of robust dialogue policies. |
SMART: Scalable Multilingual Approach for a Robust TOD System (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing TOD frameworks face significant challenges in handling unstructured information, providing multilingual support, and engaging proactively. |
| Approach: | They propose a novel TOD framework that combines traditional pipeline elements with modern agent-based approaches and features a simplified dialogue state, intelligent clarification mechanisms, and a unified natural language generation component that eliminates response redundancy. |
| Outcome: | The proposed framework outperforms baseline systems across key metrics and integrates in an e-commerce store. |
Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model (2025.acl-long)
Copied to clipboard
Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tür, Gokhan Tur
| Challenge: | Large Language Models (LLMs) with API-calling capabilities enabled building effective Language Agents (LA) current approaches excel in one domain but underperform in the other. |
| Approach: | They propose a unified approach that integrates both conversational and agentic capabilities. |
| Outcome: | The proposed model outperforms top domain-specific models across three benchmarks. |
Rethinking Task-Oriented Dialogue Systems: From Complex Modularity to Zero-Shot Autonomous Agent (2024.acl-long)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems are designed to be composed of several functional modules, but lacks a general-purpose instruction-following language model. |
| Approach: | They propose a fully zero-shot autonomous TOD agent that leverages a general-purpose instruction-following language model to decide what to do at each dialogue turn. |
| Outcome: | The proposed agent can perform tasks in real-life scenarios with a general-purpose instruction-following language model. |
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
AgentMaster: A Multi-Agent Conversational Framework Using A2A and MCP Protocols for Multimodal Information Retrieval and Analysis (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Recent advances in AI focus on multi-agent systems (MAS) that can be integrated with Large Language Models (LLMs) but current systems still face challenges of inter-agency communication, coordination, and interaction with heterogeneous tools and resources. |
| Approach: | They propose a modular multi-protocol MAS framework with self-implemented A2A and MCP . the framework supports natural language interaction without prior technical expertise . |
| Outcome: | The proposed framework supports natural language interaction without prior technical expertise and responds to multimodal queries for tasks including information retrieval, question answering, and image analysis. |