| Challenge: | a new interactive plotting agent is available for programming with natural language . the interactive aspect allows users to manipulate plots using natural language instructions. |
| Approach: | They propose an interactive natural language interface for plotting that maps language to plot updates. |
| Outcome: | The proposed system maps language to plot updates within an interactive programming environment. |
Similar Papers
ChartDialogs: Plotting from Natural Language Instructions (2020.acl-main)
Copied to clipboard
| Challenge: | a new dataset of conversational plotting agents is developed to facilitate the development of such agents. |
| Approach: | They propose a dataset that contains over 15,000 dialog turns from matplotlib's most popular plotting library. |
| Outcome: | The proposed system achieves 61% plotting accuracy, compared to the previous method. |
The Why and The How: A Survey on Natural Language Interaction in Visualization (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent research shows that different forms of natural language-based interaction prove suitable to support users in accomplishing various visualization tasks. |
| Approach: | They propose a taxonomy of visualization tasks and a classification system to illustrate the state-of-the-art of natural language-based interaction in visualization. |
| Outcome: | The proposed model can support annotations, recommendations, explanations, and documentation tasks. |
IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Language models have demonstrated their capabilities in storyline creation and human-like character role-playing. |
| Approach: | They propose a director-actor coordinate agent framework that generates drama scripts . framework allows actors to role-play their characters while maintaining plot development . |
| Outcome: | The proposed framework generates drama scripts from a drama plot outline and human actors can play their characters. |
DePlot: One-shot visual language reasoning by plot-to-table translation (2023.findings-acl)
Copied to clipboard
Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun
| Challenge: | Existing models for visual language reasoning require tens of thousands of training examples and their reasoning capabilities are limited. |
| Approach: | They propose a one-shot solution to visual language reasoning by combining plot-to-text translation and reasoning over the translated text into a modality conversion module. |
| Outcome: | The proposed method improves on human-written queries on plots and charts compared with a fine-tuned SOTA model on human queries. |
Interactive Classification by Asking Informative Questions (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for intent classification rely on a single user input and do not interact with the user to reduce ambiguity and improve the final prediction. |
| Approach: | They propose a limited form of interaction to natural language intent classification . they add binary or multi-choice questions to the system to ask missing information . |
| Outcome: | The proposed method can be bootstrapped without interaction data and is scalable to two domains. |
RolePlot: A Systematic Framework for Evaluating and Enhancing the Plot-Progression Capabilities of Role-Playing Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research has focused on role-playing agents’ ability to portray specified characters, but their ability to advance the plot requires substantial improvements to deliver more engaging interaction. |
| Approach: | They propose a role-playing framework to evaluate and enhance the plot-progression capabilities of role-players. |
| Outcome: | The proposed framework improves RPAs’ ability to time plot developments and yields a significant increase in conversation turns and sustained higher arousal levels. |
An Information-Providing Closed-Domain Human-Agent Interaction Corpus (L18-1)
Copied to clipboard
| Challenge: | a human-agent interaction corpus is a corpus of conversations between a user and an embodied conversational agent operated by a wizard of oz . data collected to create a 'corpus' with unexpected situations, such as misunderstandings, false information, and interruptions. |
| Approach: | They propose a public corpus for Human-Agent Interaction where the agent is controlled by a Wizard of Oz. |
| Outcome: | The proposed corpus is based on 15 conversations between users and a wizard of Oz agent . the data are used to create a corpus with unexpected situations, such as misunderstandings, false information, and interruptions. |
AIPOM: Agent-aware Interactive Planning for Multi-Agent Systems (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Large language models (LLMs) are being used for planning in orchestrated multi-agent systems . existing LLMs fall short of human expectations and lack effective mechanisms for users to inspect, understand, and control their behaviors. |
| Approach: | They propose a system supporting human-in-the-loop planning through conversational and graph-based interfaces. |
| Outcome: | AIPOM enables users to transparently inspect, refine, and collaboratively guide LLM-generated plans, significantly enhancing user control and trust in multi-agent workflows. |
GUI Agents: A Survey (2025.findings-acl)
Copied to clipboard
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt
| Challenge: | Large Foundation Models (LFMs) have transformed the landscape of AI research and day-to-day life. |
| Approach: | They propose a framework that delineates GUI agents' perception, reasoning, planning, and acting capabilities. |
| Outcome: | The proposed framework delineates their perception, reasoning, planning, and acting capabilities. |
Improving Natural Language Interaction with Robots Using Advice (N19-1)
Copied to clipboard
| Challenge: | Recent studies focus on learning models for physically grounded language understanding tasks such as the blocks world domain. |
| Approach: | They propose a protocol for including advice, high-level observations about the task, which can help constrain the agent’s prediction. |
| Outcome: | The proposed approach can be extended to include advice, high-level observations about the task, and reduce the effort involved in supplying the advice. |