Exploring Language Model’s Code Generation Ability with Auxiliary Functions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Auxiliary function is a useful component to improve language model’s code generation ability, but a systematic exploration of how they affect has yet to be done. |
| Approach: | They construct a human-crafted evaluation set which contains examples of two functions where one function assists the other to examine their ability in a multifaceted way. |
| Outcome: | The proposed model is underutilized to call the auxiliary function, suggesting future directions to enhance their implementation by eliciting the supplementary function call ability encoded in the models. |
Similar Papers
Eliciting Instruction-tuned Code Language Models’ Capabilities to Utilize Auxiliary Function for Code Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using auxiliary functions to implement functions is important for instruction-tuned models because it reduces the implementation difficulty of a target function compared to implementing them from scratch. |
| Approach: | They propose several ways to provide auxiliary functions to the models by adding them to the query or providing a response prefix to incorporate the ability to utilize auxiliary function with the instruction following capability. |
| Outcome: | The proposed models outperform the recent powerful language models, gpt-4o, in the code generation task. |
Exploring Data Augmentation for Code Generation Tasks (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have impacted how models are trained for programming language tasks. |
| Approach: | They propose to use augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% respectively. |
| Outcome: | The proposed methods improve translation and summarization by 6.9% and 7.5% respectively. |
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021.naacl-main)
Copied to clipboard
| Challenge: | Empirical results show that today’s language models struggle at TuringAdvice . language models are getting ever-larger, and are being trained on ever-increasing quantities of text . |
| Approach: | They propose a task task that requires models to generate helpful advice in natural language. |
| Outcome: | The proposed model outperforms even multibillion parameter models on 600k in-domain training examples. |
Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models have been claimed to acquire certain capabilities without having been specifically trained on them. |
| Approach: | They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments. |
| Outcome: | The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge. |
HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for code generation tasks are inadequate, but performance declines on self-invoking tasks. |
| Approach: | They propose a general recipe for generating more challenging versions of existing benchmarks . they propose to use instruction-tuned models to evaluate LLMs on self-invoking code generation tasks . |
| Outcome: | The proposed model improves on humanEval and MBPP but on self-invoking code generation tasks. |
NLP+Code: Code Intelligence in Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | Language models have shown impressive abilities in a range of natural language processing tasks. |
| Approach: | This tutorial will provide an overview of the latest advances in natural language processing . it will provide preliminaries of training foundation models on code and their common practices . |
| Outcome: | This tutorial aims to provide an overview of recent advances in code modeling . it provides preliminaries of training foundation models on code and their common practices . |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work examines the cognitive capabilities of language models through psychological tests designed for humans. |
| Approach: | They propose to use human-like cognitive capabilities to evaluate language models . they propose to augment language models with better listeners to improve their performance . |
| Outcome: | The proposed method boosts language models with better models of the listener and improves them. |
ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for optimizing program efficiency improve performance measured by execution time, but they often come at the cost of severely decreasing the functional correctness. |
| Approach: | They propose a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. |
| Outcome: | The proposed approach improves performance while maintaining correctness while adding execution information. |