Yuhao Zhang, Shiqi Wang, Haifeng Qian, Zijian Wang, Mingyue Shang, Linbo Liu, Sanjay Krishna Gouda, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras
| Challenge: | Existing research efforts to improve code generation models are inadequate . code generation model performance is degraded under small perturbations . |
| Approach: | They propose a framework to improve the robustness of code generation models by generalizing code perturbations to enrich training data and enabling various robust training strategies. |
| Outcome: | The proposed framework increases pass rates and robustness drop rate against code-syntax perturbations. |
Similar Papers
ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)
Copied to clipboard
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang
| Challenge: | Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area. |
| Approach: | They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. |
| Outcome: | The proposed model performs better on human annotators and on SOTA models with human annnotators. |
Evaluating and Enhancing the Robustness of Code Pre-trained Models through Structure-Aware Adversarial Samples Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained code models have made significant strides in the field of neural code intelligence, but they are susceptible to adversarial attacks that subtly modify the input sequence and can impair generalization. |
| Approach: | They propose a set of novel robustness evaluation methods based on the intrinsic structure of the code to explore the impact of imperceptible perturbation. |
| Outcome: | The proposed methods have demonstrated their effectiveness across a wide range of models and tasks, and are able to predict the performance of perturbed models. |
Generative Adversarial Training with Perturbed Token Detection for Model Robustness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial training methods use discrete tokens to deceive models . current approaches use embeddings, whereas actual text-based training uses discrete text tokens. |
| Approach: | They propose a framework that integrates gradient-based learning, adversarial example generation and perturbed token detection to enhance adversariarial robustness. |
| Outcome: | The proposed framework surpasses the state-of-the-art results of ChatGPT by 10% in average accuracy. |
Extracting the Essence and Discarding the Dross: Enhancing Code Generation with Contrastive Execution Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | erroneous code generation methods amalgamate feedback and correct code as target sentences . a new approach to code generation with feedback is needed to improve model performance . |
| Approach: | They propose a learning-based code generation model with execution feedback that integrates feedback and correct code as target sentences. |
| Outcome: | a new model with execution feedback shows improvements in generating accurate code and understanding error correction. |
CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text. |
| Approach: | They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language. |
| Outcome: | The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models. |
Exploring Data Augmentation for Code Generation Tasks (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have impacted how models are trained for programming language tasks. |
| Approach: | They propose to use augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% respectively. |
| Outcome: | The proposed methods improve translation and summarization by 6.9% and 7.5% respectively. |
FrontCoder: Scaling Visual Fidelity in Front-End Code Generation (2026.findings-acl)
Copied to clipboard
Jun Feng, Jian Yang, Wei Zhang, Jing Wang, Keyi Chen, Xiaokun Yang, Weicheng Gu, Yihang Lou, Yan Bai, Xianglong Liu
| Challenge: | Existing work on front-end code generation fails to provide visual fidelity and rendering quality for front- end developers. |
| Approach: | They propose a three-stage pipeline to enhance front-end code generation capabilities in LLMs . they use synthetic data, quality-controlled supervised fine-tuning, and reinforcement learning . |
| Outcome: | The proposed model achieves competitive performance with frontier models while maintaining generation efficiency. |
A Simple, Yet Effective Approach to Finding Biases in Code Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that large language models can generate code on par with humans . however, data-driven approaches may not be sufficient for acquiring reasoning skills . |
| Approach: | They propose a framework that automatically identifies subtle cues a code generation model might exploit . they propose an automated intervention mechanism reminiscent of adversarial testing . |
| Outcome: | The proposed framework can be used as a data transformation technique during fine-tuning, acting as reversal strategy. |
Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce model's reliance on bias features ignore the learnability of these features. |
| Approach: | They propose to reduce models' reliance on bias features by first training models with fixed low-capacity models which ignore the learnability of the bias features. |
| Outcome: | The proposed models can perform better on out-of-distribution datasets than baseline models with a more sophisticated model design. |
CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial text generation approaches can lead to generation lacking diversity or fluency, whereas perturbing in the intermediate representation space can lead a model to generate generations that are not related to the input. |
| Approach: | They propose to generate adversarial texts through controllable attributes that are known to be invariant to task labels. |
| Outcome: | The proposed model generates more diverse and fluent adversarial examples, compared to existing approaches, and is more robust against model re-training and different model architectures. |