Multi-Step Generation of Test Specifications using Large Language Models for System-Level Requirements (2025.acl-industry)
Copied to clipboard
| Challenge: | System-level testing is a critical phase in the development of large, safety-dependent systems, such as those in the automotive industry. |
| Approach: | They propose an AI-powered assistant to aid users in creating test specifications for system-level requirements. |
| Outcome: | The proposed system reduces the effort required to derive test specifications by 30% in ROUGE-L. |
Similar Papers
ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Despite the critical role of software requirements, these criteria have not been studied actively in previous code generation works. |
| Approach: | They propose a framework that leverages in-context learning to organize and extrapolate unexpressed requirements from textual descriptions. |
| Outcome: | The proposed framework generates functional requirements from textual descriptions and extrapolates unexpressed requirements from them. |
CodeSim: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides in code generation and problem solving. |
| Approach: | They propose a multi-agent code generation framework that integrates human-like perception to address the stages of program synthesis. |
| Outcome: | The proposed framework achieves state-of-the-art (pass@1) results and shows potential for even greater enhancement when cascaded with external debuggers. |
Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents (2026.eacl-long)
Copied to clipboard
| Challenge: | Robust, developer-friendly evaluation remains a bottleneck. |
| Approach: | They propose a meta-agent that combines static code analysis, developer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback. |
| Outcome: | The agent-testing agent (ATA) surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20–30 minutes versus ten-annotator rounds that took days. |
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications (2026.acl-long)
Copied to clipboard
| Challenge: | Existing safety evaluation paradigms rely on constructed benchmarks or dynamic red-teaming to probe potential vulnerabilities. |
| Approach: | They propose a framework that integrates specification-based software testing with AI safety. |
| Outcome: | The proposed framework achieves higher coverage and attack success counts compared to baselines. |
Formally Specifying the Intended Behavior of the Program: LLM-Driven Neuro-Symbolic Program Specification Synthesis (2026.acl-demo)
Copied to clipboard
Cheng Wen, Hu Junjie, YiKun Hu, Jie Su, Bin Yu, Dugang Liu, Zhiwu Xu, Weidi Sun, Shengchao Qin, Cong Tian
| Challenge: | Formal verification typically requires developers to write detailed formal specifications . a formal verification system that generates candidate specifications is costly and error-prone . |
| Approach: | They propose an LLM-driven neuro-symbolic demonstration system that reframes specification writing as constrained structured synthesis. |
| Outcome: | The proposed system reduces hallucinations and produces proof-ready annotations. |
Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing agentic training data are narrow in task variety and easily solved . real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. |
| Approach: | They propose a framework that synthesizes diverse tool-use training data and simulates complete environments. |
| Outcome: | The proposed framework synthesizes diverse tool-use training data and simulates complete environments. |
TESTEVAL: Benchmarking Large Language Models for Test Case Generation (2025.findings-naacl)
Copied to clipboard
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma
| Challenge: | Existing methods to generate test cases using large language models are limited in their ability to generate unit test cases. |
| Approach: | They propose a test case generation benchmark that uses large language models to generate unit test cases. |
| Outcome: | The proposed test case generation benchmarks compare LLMs with commercial and open-source LLM platforms and find that they lack the ability to comprehend program logic and execution paths. |
Can Intelligent Agents Revolutionize Scale Generation? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing measurement scales require extensive manual labor and require extensive validation and validation. |
| Approach: | They propose a multi-agent framework that automates scale development by leveraging collaborative AI agents. |
| Outcome: | The proposed framework automates scale development while maintaining rigorous quality standards. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024.emnlp-industry)
Copied to clipboard
Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang
| Challenge: | Large language models have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools. |
| Approach: | They propose a framework to enhance the task planning and tool usage abilities of LLMs in industrial systems. |
| Outcome: | The proposed framework enhances the task planning and tool usage abilities of LLM-based agents in industrial systems. |