Can docstring reformulation with an LLM improve code generation? (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches focus on training, fine-tuning or prompting LLMs to generate better outputs given the same input. |
| Approach: | They propose to optimize part of the input, the docstring, via reformulation with an LLM to improve code generation. |
| Outcome: | The proposed methods improve code generation on the original HumanEval benchmark and multiple curated variants on the same input. |
Similar Papers
DocCGen: Document-based Controlled Code Generation (2024.emnlp-main)
Copied to clipboard
Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Kumar, Pushpak Bhattacharyya
| Challenge: | Large language models (LLMs) produce state-of-the-art performance on natural language to code generation for resource-rich general-purpose languages like C++, Java, and Python. |
| Approach: | They propose a framework that breaks the NL-to-Code generation task into two steps . they use library documentation to detect the correct libraries and schema rules extracted from the documentation to constrain the decoding . |
| Outcome: | The proposed framework improves different sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code. |
What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation (2026.acl-long)
Copied to clipboard
Shaomu Tan, Dawei Zhu, Ke Tran, Michael Denkowski, Sony Trenous, Leonardo F. R. Ribeiro, Bill Byrne, Felix Hieber
| Challenge: | Large language models (LLMs) have made document-level machine translation increasingly practical, enabled by long-context modeling and strong generation quality. |
| Approach: | They propose to use document-level MT followed by segment-level refinement to find the strongest and most stable improvements across six LLMs and seven language pairs. |
| Outcome: | The proposed method outperforms error-specific prompting and evaluate-then-refine schemes in document-level translation. |
Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs? (2025.findings-acl)
Copied to clipboard
Qingyuan Liang, Zhao Zhang, Zeyu Sun, Zheng Lin, Qi Luo, Xiao Yueyi, Yizhou Chen, Yuqun Zhang, Haotian Zhang, Lu Zhang, Chenbin Chenbin, Yingfei Xiong
| Challenge: | Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance. |
| Approach: | They develop a series of billion-scale grammar-based code representations that incorporate grammar rules into the code generation process. |
| Outcome: | Experiments on HumanEval and MBPP show that grammar-based representations reduce syntax errors and improve performance even in billion-scale models. |
Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs (2025.naacl-srw)
Copied to clipboard
| Challenge: | Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development. |
| Approach: | They propose to use Reinforcement Learning and Direct Preference Optimization to fine-tune code-generating Large Language Models (LLMs) by enhancing the training data with symbolic execution techniques. |
| Outcome: | The proposed model improves on the CodeRL benchmark and shows that it is more accurate and objective than the baseline model. |
Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints (2025.acl-short)
Copied to clipboard
| Challenge: | Semantic Parsing improves performance of smaller models, but it is unclear whether it extends similarly to large language models. |
| Approach: | They propose a prompting approach that embeds semantic hints within the prompt to improve LLM performance. |
| Outcome: | The proposed approach improves LLMs’ performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities. |
I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has demonstrated that a large language model (LLM) can generate training data for another LLM, or for creating supplementary training materials, such as rationales. |
| Approach: | They conduct an in-depth investigation to understand why fine-tuning an LLM with responses generated by a LLM often yields better results than using responses generated from humans. |
| Outcome: | The proposed approach can be used to transfer knowledge from a larger model to a smaller one, or for creating supplementary training materials, such as rationales. |
Testing the Effect of Code Documentation on Large Language Model Code Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive abilities in recent years with regards to code generation and understanding. |
| Approach: | They propose to provide an LLM with "incorrect" documentation that can greatly hinder code understanding, while incomplete or missing documentation does not seem to significantly affect an LRM's ability to understand code. |
| Outcome: | The proposed model can generate and understand code in a language with high documentation quality while lacking documentation does not significantly affect the ability to understand code. |
ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for optimizing program efficiency improve performance measured by execution time, but they often come at the cost of severely decreasing the functional correctness. |
| Approach: | They propose a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. |
| Outcome: | The proposed approach improves performance while maintaining correctness while adding execution information. |
CodecLM: Aligning Language Models with Tailored Synthetic Data (2024.findings-naacl)
Copied to clipboard
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
| Challenge: | Recent work on generating diverse instructions and applying LLM to increase instruction complexity neglects downstream use cases. |
| Approach: | They propose a framework for generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. |
| Outcome: | Experiments on four open-domain instruction using the proposed framework validate the effectiveness of CodecLM over the current state-of-the-art. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |