LUMINA: Long-horizon Understanding for Multi-turn Interactive Agents (2026.findings-acl)
Copied to clipboard
Amin Rakhsha, Thomas Hehn, Pietro Mazzaglia, Fabio Valerio Massoli, Arash Behboodi, Tribhuvanesh Orekondy
| Challenge: | Large language models struggle on multi-turn, long-horizon agentic problems that require skills such as planning, state tracking, and long context processing. |
| Approach: | They propose an oracle counterfactual framework for multi-turn problems that asks: how would an agent perform if it could leverage an or acle to execute a specific skill? |
| Outcome: | The proposed framework allows for precise oracle interventions without confounding effects present in real-world benchmarks. |
Similar Papers
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)
Copied to clipboard
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang
| Challenge: | Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics. |
| Approach: | They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks. |
| Outcome: | The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring. |
FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments (2026.findings-acl)
Copied to clipboard
Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral
| Challenge: | Large Language Models are being increasingly deployed as decision-making core of autonomous agents . however, in conversational benchmarks, these agents fail due to the cascading effects of incorrect decision- making . |
| Approach: | They propose a framework that analyzes failure trajectories from baseline agents to identify most prevalent errors. |
| Outcome: | Experiments show that the framework improves performance over open-source LLMs . the framework can be used to build reliable, multi-turn tool-use agents . |
Towards Effective and Efficient Multi-Agent Language Model Systems: Foundations, Prospects, and Applications (2026.acl-tutorials)
Copied to clipboard
| Challenge: | Multi-agent systems powered by large language models still face challenges . tutorial focuses on three core components to build effective and efficient systems . |
| Approach: | This tutorial introduces recent advances in building effective and efficient multi-agent LLM systems . it focuses on three core components: model distillation, dynamic routing, memory- and compute efficient serving . |
| Outcome: | This tutorial introduces state-of-the-art techniques for building efficient and efficient multi-agent LLM systems . it covers coordination and communication among agents, crucial for collective performance . |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
Experience-driven Multi-turn Reinforcement Learning for GUI Agents (2026.acl-long)
Copied to clipboard
Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, Yueting Zhuang
| Challenge: | GUI agents have demonstrated remarkable progress in automating complex user interface interactions . training such agents for long-horizon tasks remains challenging due to limited rewards and prohibitive costs. |
| Approach: | They propose a method that leverages expert trajectories as environment experiences for on-policy multi-turn training. |
| Outcome: | The proposed method achieves significant gains over the base model with 1K public trajectories as RL experiences . it achieves competitive performance against strong baselines such as UI-TARS-7B and GPT-4o . |
SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities, but their application to complex, multi-step, and long-horizon tasks remains challenging. |
| Approach: | They propose a framework that provides a finer-grained advantage assignment derived solely from outcome rewards. |
| Outcome: | The proposed framework provides a finer-grained advantage assignment, derived solely from outcome rewards. |
Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration (2026.findings-acl)
Copied to clipboard
Linzhuang Sun, Tianyu Guo, Hao Liang, Ruitong Liu, Yuying Li, Qifeng Cai, Jingxuan Wei, Yuchen Wu, Bihui Yu, Xiangxiang Zhang, Wentao Zhang, Bin Cui
| Challenge: | Structured Query Language (SQL) is the cornerstone for data-driven decision-making. |
| Approach: | They propose a benchmark to rigorously evaluate Large Language Models within a dynamic interaction framework. |
| Outcome: | The proposed benchmark aims to rigorously evaluate LLMs within a dynamic interaction framework. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
SQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL (2026.acl-long)
Copied to clipboard
Harper Hua, Zhen Han, Zhengyuan Shen, Meng-Chieh Lee, Sheng Guan, Qi Zhu, Sullam Jeoung, Yueyan Chen, Yunfei Bai, Shuai Wang, Vassilis N. Ioannidis, Huzefa Rangwala
| Challenge: | Recent large language models (LLMs) have significantly improved Text-to-SQL generation, but a gap remains between AI systems and human experts on challenging benchmarks such as BIRD-Sql. |
| Approach: | They propose a multi-turn reinforcement learning agentic framework for Text-to-SQL that uses execution feedback to iteratively refine its predictions. |
| Outcome: | The proposed framework outperforms proprietary systems on 7B and 14B models by **5% on average, underscoring the effectiveness of interactive, agentic workflows for robust Text-to-SQL generation. |