Beyond Superficial Tests: Adversarial Refinement for Reliable Property-Based Testing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, yet their application to Property-Based Testing (PBT) remains fraught with a superficiality gap. |
| Approach: | They propose an agentic framework that hardens software properties through Adversarial Refinement. |
| Outcome: | a new framework hardens software properties through Adversarial Refinement that detects and fixes bugs in top-tier libraries. |
Similar Papers
CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Models often lack coverage for subtle corner cases . a substantial amount of effort has been applied to address this challenge . |
| Approach: | They propose a framework that generates adversarial test cases that expose latent vulnerabilities in code submissions. |
| Outcome: | The proposed framework improves the True Negative Rate (TNR) of existing datasets and generates superior adversarial cases on liveCodeBench. |
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications (2026.acl-long)
Copied to clipboard
| Challenge: | Existing safety evaluation paradigms rely on constructed benchmarks or dynamic red-teaming to probe potential vulnerabilities. |
| Approach: | They propose a framework that integrates specification-based software testing with AI safety. |
| Outcome: | The proposed framework achieves higher coverage and attack success counts compared to baselines. |
Reliability Testing for Natural Language Processing Systems (2021.acl-long)
Copied to clipboard
| Challenge: | a lack of rigorous testing and ML implicit assumption of identical training and testing distributions may result in systems that discriminate against minorities. |
| Approach: | They argue that reliability testing is needed to address the issue of demographics . they argue that adversarial attacks can be reframed for this goal . |
| Outcome: | The proposed framework will enable rigorous and targeted testing and aid in the enactment and enforcement of industry standards. |
PropTest: Automatic Property Testing for Improved Visual Programming (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Programming is an alternative to end-to-end black-box visual reasoning models. |
| Approach: | They propose a visual programming strategy that leverages Large Language Models to generate the logic of a program in the form of its source code. |
| Outcome: | The proposed method improves ViperGPT on visual question answering and referring expression comprehension with an LLM. |
Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on this topic focus on the robustness of specific detectors or particular attack methods. |
| Approach: | They stress test the detectors’ robustness to malicious attacks under realistic scenarios using LLMs and metric-based detectors. |
| Outcome: | The proposed methods are based on a set of LLM-based models and their performance is compared under different budget levels. |
PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-SQL systems struggle with deep contextual understanding . |
| Approach: | They propose a framework that provides a tool to help query databases with deeper contextual understanding . they propose two components that iteratively generate probing queries and verify queries . |
| Outcome: | Experiments show PV-SQL outperforms the best text-to-SqL baseline by 5% execution accuracy and 20.8% valid efficiency score while consuming fewer tokens. |
How Adversarial Environments Mislead Agentic AI? (2026.findings-acl)
Copied to clipboard
| Challenge: | Current evaluations benchmark capability in benign settings, but never "what if the tools lie" we formalize this vulnerability as Adversarial Environmental Injection (AEI) AEI constitutes environmental deception by constructing a "fake world" of poisoned search results . |
| Approach: | They propose an attack model where adversaries compromise tool outputs to deceive agents. |
| Outcome: | The proposed model exploits a trust gap between tool outputs and actual exposure to adversaries. |
Toward Automated Robustness Evaluation of Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Yutao Hou, Zeguan Xiao, Fei Yu, Yihan Jiang, Ma Shuguang, Zhaoqian Dai, Hailiang Huang, Yun Chen, Guanhua Chen
| Challenge: | Existing robustness evaluations rely on hand-crafted templates or a limited set of perturbation rules, resulting in model failure. |
| Approach: | They propose a framework inspired by software stress testing that generates adversarial variants via a multi-round rewrite-verify loop, ensuring semantic consistency while successfully inducing model failure. |
| Outcome: | The proposed framework generates adversarial variants dynamically for each LLM, minimizing the risk of data contamination. |
False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in Large Language Models’ internal representations. |
| Approach: | They propose to use probing-based methods to study separability of malicious and benign inputs in LLMs' internal representations to detect harmful and benign content. |
| Outcome: | The proposed methods show that they learn superficial patterns rather than semantic harmfulness. |
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? (2026.findings-acl)
Copied to clipboard
Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang, Zhenya Huang
| Challenge: | Evaluating software engineering capabilities is a core component of large language models (LLMs). |
| Approach: | They propose a benchmark to evaluate LLM-generated test suites that introduces mutated solutions that attempt to "fool" them. |
| Outcome: | The proposed test suites are based on 2,636 mutated variants derived from 800 original instances and include a multilingual subset spanning nine programming languages. |