Code Representation Pre-training with Complements from Program Executions (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing languages have syntactic representations of code to improve code intelligence, but they are difficult to learn from code. |
| Approach: | They propose to embed dynamic information of programs revealed by their test cases into feature representations of code as complements. |
| Outcome: | The proposed method yields 6%/19% mAP improvements over its masked language modeling counterparts. |
Similar Papers
Understanding Programs by Exploiting (Fuzzing) Test Cases (2023.findings-acl)
Copied to clipboard
| Challenge: | Semantic understanding of programs has attracted great attention in the community . large language models (LLMs) are capable of learning contextual information from data at scale . |
| Approach: | They propose to incorporate a relationship between inputs and possible outputs into learning for achieving a deeper semantic understanding of programs. |
| Outcome: | The proposed method outperforms current state-of-the-art on two programming tasks and outperformed current state of the art by large margins. |
Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
Discovering Representation Sprachbund For Multilingual Pre-Training (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models perform poorly on many languages and cross-lingual tasks due to typological differences and contradictions between some languages. |
| Approach: | They propose to pre-train multilingual pre-trained models to handle cross-lingual tasks in one model. |
| Outcome: | The proposed model improves performance on cross-lingual tasks compared to baselines on multiple languages . |
Benchmarking Language Models for Code Syntax Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models capture the syntactic rules of natural languages without fine-tuning on syntax understanding tasks. |
| Approach: | They propose a benchmarking test to compare pre-trained language models with a large-scale dataset of programs annotated with syntactic relationships in their corresponding abstract syntax trees. |
| Outcome: | The proposed model fails to match baselines based on positional offsets and keywords. |
Contrastive Code Representation Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work learns contextual representations of source code by reconstructing tokens from their context. |
| Approach: | They propose a contrastive pre-training task that learns code functionality, not form . they propose scalable compilers that can generate variants of a program . |
| Outcome: | The proposed task outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. |
CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on code representation learning, which aims to represent the semantics of source code into distributed vectors. |
| Approach: | They propose to integrate different views with the natural-language description of source code into a unified framework with Multi-View contrastive Pre-training. |
| Outcome: | The proposed model outperforms state-of-the-art models on three downstream tasks over five datasets. |
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models for Natural Languages (NL) like BERT and GPT have been shown to transfer well to Programming Languages. |
| Approach: | They propose a unified pre-trained encoder-decoder Transformer model that leverages the code semantics conveyed from the developer-assigned identifiers. |
| Outcome: | The proposed model outperforms existing models on understanding and generation tasks and can capture semantic information from code. |
Unified Pre-training for Program Understanding and Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | PLUG is a programming language that is used for programming and language understanding and generation tasks. |
| Approach: | They propose a sequence-to-sequence model that performs a broad spectrum of program and language understanding and generation tasks. |
| Outcome: | The proposed model outperforms or rivals state-of-the-art models on code summarization, code generation, and code translation tasks in seven programming languages. |
UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models for programming languages have demonstrated great success on code intelligence . however, such pre-tried models are sub-optimal for auto-regressive tasks . |
| Approach: | They propose a unified cross-modal pre-trained model for programming language that leverages cross-module contents like AST and code comment to enhance code representation. |
| Outcome: | The proposed model achieves state-of-the-art on most code-related tasks and compares with existing models on zero-shot code-to-code search. |
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)
Copied to clipboard
| Challenge: | NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge . |
| Approach: | They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code . |
| Outcome: | The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages. |