AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
Similar Papers
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition (2026.findings-acl)
Copied to clipboard
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, Dongzhan Zhou
| Challenge: | Large language models have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. |
| Approach: | They propose a benchmark for evaluating large language models on a sufficient set of scientific discovery sub-tasks. |
| Outcome: | The proposed framework extracts critical components from papers across 12 disciplines with expert validation confirming its accuracy. |
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation (2025.findings-acl)
Copied to clipboard
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, Peter Clark
| Challenge: | Automated scientific discovery (ASD) systems are limited in their evaluation of software artifacts and large volumes of research artifs are typically evaluated using conference-style paper review with limited evaluation of code. |
| Approach: | They propose a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain. |
| Outcome: | The proposed system returns 19 discoveries on machine-generated ideas in the domain of agents and virtual environments. |
CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Automated Scientific Discovery (ASD) systems rely on parametric knowledge to generate and run code-based experiments. |
| Approach: | They propose a system that distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples. |
| Outcome: | The proposed system produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. |
Quest2DataAgent: Automating End-to-End Scientific Data Collection (2025.emnlp-demos)
Copied to clipboard
Tianyu Yang, Yuhan Liu, Sobin Alosious, Ethan A. Brown, Jason R. Rohr, Tengfei Luo, Xiangliang Zhang
| Challenge: | Existing approaches for data collection are labor-intensive and dependent on domain expertise. |
| Approach: | They propose a general-purpose multi-agent framework for automating scientific data collection workflows. |
| Outcome: | The proposed framework improves data relevance, usability, and time efficiency over existing methods. |
MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows (2025.findings-naacl)
Copied to clipboard
Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, Qiaozhu Mei
| Challenge: | Scientific innovation is driven by detailed workflows, which include critical steps such as contextualizing literature, generating ideas, validating ideas, and planning new research. |
| Approach: | They propose to use large language models to extract five key aspects from scientific publications to optimize scientific workflows. |
| Outcome: | The proposed dataset includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. |
InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents (2026.findings-acl)
Copied to clipboard
Zhenghao Zhu, Yuanfeng Song, Xing Chen, Chengzhong Liu, Cui Yakun, Caleb Chen Cao, Sirui Han, Yike Guo
| Challenge: | Existing frameworks for data analysis and insight exploration are lacking in terms of benchmarks . existing frameworks suffer from format inconsistencies, poorly conceived objectives, and redundant insights. |
| Approach: | They propose a data-curation pipeline to construct a new dataset named InsightEval. |
| Outcome: | The proposed benchmarks highlight prevailing challenges in automated insight discovery and raise key findings to guide future research. |
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (2025.acl-long)
Copied to clipboard
Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Xu Yinghui, Wei Chu, Zili Wang
| Challenge: | Code LLMs lack reproducible data pipelines and training protocols for reproducible advancements in code intelligence. |
| Approach: | They propose a top-tier code LLM that releases model weights and inference code . reproducible data pipelines, rigorous experimental ablation results and training protocols are included . |
| Outcome: | The proposed model achieves comparable performance to leading models and serves as an "open cookbook" reproducible training data, rigorous experimental ablation results, and detailed training protocols are also included in the model. |
DiscoverGPT: Multi-task Fine-tuning Large Language Model for Related Table Discovery (2025.findings-naacl)
Copied to clipboard
Xuming Hu, Xiao Qin, Chuan Lei, Asterios Katsifodimos, Zhengyuan Shen, Balasubramaniam Srinivasan, Huzefa Rangwala
| Challenge: | Existing methods to learn and evaluate the table semantic relatedness of tabular data are based on pretrain-and-finetune paradigms. |
| Approach: | They propose a multi-task fine-tuning framework that holistically discovers and leverages the intricate relationships among the supervisions to optimize the performance on the data discovery task. |
| Outcome: | The proposed framework outperforms the best performing baseline by up to 7% in F1 score. |
DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu
| Challenge: | DA-Code is a code generation benchmark designed to assess LLMs on agent-based data science tasks. |
| Approach: | They propose a code generation benchmark specifically designed for LLMs on agent-based data science tasks. |
| Outcome: | The benchmark performs better than existing frameworks, but lacks accuracy . it is based on real-world data, and includes examples that cover a wide range of tasks . |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |