InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents (2026.findings-acl)
Copied to clipboard
Zhenghao Zhu, Yuanfeng Song, Xing Chen, Chengzhong Liu, Cui Yakun, Caleb Chen Cao, Sirui Han, Yike Guo
| Challenge: | Existing frameworks for data analysis and insight exploration are lacking in terms of benchmarks . existing frameworks suffer from format inconsistencies, poorly conceived objectives, and redundant insights. |
| Approach: | They propose a data-curation pipeline to construct a new dataset named InsightEval. |
| Outcome: | The proposed benchmarks highlight prevailing challenges in automated insight discovery and raise key findings to guide future research. |
Similar Papers
Benchmarking Data Science Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as promising data science aids, assisting humans in data analysis and processing. |
| Approach: | They propose an evaluation paradigm and benchmarks that assess the performance of data science agents throughout the entire data science lifecycle. |
| Outcome: | The proposed evaluation paradigm streamlines dataset preparation, improves coverage, and expands benchmarking comprehensiveness. |
UniDataBench: Evaluating Data Analytics Agents Across Structured and Unstructured Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks do not assess agents’ capabilities across data types . Existing tools only evaluate agents' ability to extract reasonable insights across data formats. |
| Approach: | They propose a multi-source benchmark to evaluate the performance of data analytics agents in handling diverse data sources. |
| Outcome: | The proposed agent performs end-to-end analysis over diverse data sources by automatically discovering cross-source linkages, decomposing goals, and generating robust, self-correcting code to extract actionable insights. |
DataSage: Multi-agent Collaboration for Insight Discovery with External Knowledge Retrieval, Multi-role Debating, and Multi-path Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing data insight agents fail to deliver satisfactory results due to insufficient utilization of domain knowledge, shallow analytical depth, and error-prone code generation. |
| Approach: | They propose a novel multi-agent framework that incorporates external knowledge retrieval to enrich the analytical context, a multi-role debating mechanism to simulate diverse analytical perspectives and deepen analytical depth, and multi-path reasoning to improve the accuracy of the generated code and insights. |
| Outcome: | Extensive experiments on InsightBench show that DataSage outperforms existing data insight agents across all difficulty levels, improving by 7.5% and 13.9% respectively in insight-level and summary-level metrics. |
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition (2026.findings-acl)
Copied to clipboard
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
| Challenge: | Large language models have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. |
| Approach: | They propose a benchmark for evaluating large language models on a sufficient set of scientific discovery sub-tasks. |
| Outcome: | The proposed framework extracts critical components from papers across 12 disciplines with expert validation confirming its accuracy. |
InsightPilot: An LLM-Empowered Automated Data Exploration System (2023.emnlp-demo)
Copied to clipboard
| Challenge: | InsightPilot is an LLM-based, automated data exploration system designed to simplify the data exploration process. |
| Approach: | They propose an LLM-based, automated data exploration system that streamlines the data exploration process. |
| Outcome: | InsightPilot is an LLM-based, automated data exploration system . it can help users gain valuable insights from their datasets, in a case study and in nl . |
INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent (2025.acl-long)
Copied to clipboard
Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, K.p. Subbalakshmi, Jimin Huang, Lingfei Qian, Xueqing Peng, Jordan W. Suchow, Qianqian Xie
| Challenge: | Recent advances have underscored the potential of large language model (LLM)-based agents in financial decision-making. |
| Approach: | They propose to evaluate LLM agents using 13 different LLMs as backbone models across various market environments and tasks. |
| Outcome: | The proposed framework assesses the reasoning and decision-making capabilities of 13 different LLMs across various market environments and tasks. |
DataSeer: A Manager-Centric Collaborative Multi-Agent Framework with Multi-Branch Reasoning for Automated Insight Discovery (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for automated insight discovery lack contextual coherence and coverage due to single-path exploration. |
| Approach: | They propose a Manager-Centric Collaborative Framework that integrates planner and executor . it ensures cross-episode contextual coherence and allows for adaptive sub-goal generation . |
| Outcome: | The proposed framework outperforms baselines on InsightBench and Inseval. |
ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks lack the ability to automatically evaluate from users’ perspective and lack the explainability of the results of LLM agents’ code generation capabilities. |
| Approach: | They propose a new benchmark for LLM agents' automated evaluation by simulating user interaction. |
| Outcome: | The proposed benchmark can evaluate the generated projects by user interaction simulation and by code similarity through existing objective indicators. |
DataSciBench: An LLM Agent Benchmark for Data Science (2026.findings-acl)
Copied to clipboard
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, Yisong Yue
| Challenge: | Existing benchmarks focus on single task, simple evaluation metrics, and readily available ground truth (GT) DataSciBench is built on curated, natural, and challenging prompts with complex evaluation criteria and uncertain GT. |
| Approach: | They propose a benchmark for evaluating Large Language Models in data science that integrates LLM-based self-consistency and human verification to ensure accuracy. |
| Outcome: | The proposed framework outperforms open-source models in all metrics and offers rigorous insights into LLM strengths and weaknesses. |
FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-based agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. |
| Approach: | They propose a benchmark to evaluate the efficacy of workflow-guided agent planning by formalizing different formats of workflow knowledge. |
| Outcome: | The proposed benchmark aims to improve the planning reliability of LLM-based agents by incorporating external workflow-related knowledge. |