The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Open-source dataset of code-text pairs for training large language models to understand code is outperforms other datasets for code generation and understanding tasks. |
| Approach: | They propose to extract high-quality code-text pairs from a dataset of 43 million pairs . they use rules and deep learning to ensure that the code-sampled samples contain high-quality pairs a . |
| Outcome: | The Vault dataset outperforms existing models on common coding tasks . authors hope the results will propel AI research and software development forward . |
Similar Papers
XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. |
| Approach: | They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages. |
| Outcome: | The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens) |
Language Agnostic Code Embeddings (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that code language models have strong cross-lingual traits, but their multilingual representations can be dissected into a language-specific syntax component and a semantic component. |
| Approach: | They propose to isolate and eliminate language-specific components from multilingual code embeddings to improve downstream code retrieval tasks. |
| Outcome: | The proposed model improves retrieval tasks by removing language-specific components . the proposed model can be used to perform a variety of code generation tasks . |
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing code translation datasets focus on a single pair of programming languages . early software systems are developed using programming languages such as Fortran and COBOL . |
| Approach: | They propose a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. |
| Outcome: | The proposed framework supports translations between multiple programming languages and a cross-framework dataset for deep learning code across different frameworks. |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |
CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024.findings-acl)
Copied to clipboard
| Challenge: | Comprising 3,402 crafted examples, our dataset is designed for both model finetuning and standalone evaluation. |
| Approach: | They propose a dataset that provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. |
| Outcome: | The proposed dataset includes 3,402 hand-written examples and 3,121 unrefined examples. |
Optimizing Code Retrieval: High-Quality and Scalable Dataset Annotation through Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for code retrieval struggle to balance scalability and annotation quality. |
| Approach: | They propose a method that integrates functions called within the repository and information on third-party APIs to enhance the annotation context. |
| Outcome: | The proposed method improves the annotation context by incorporating functions called within the repository and information on third-party API functionalities. |
Large Language Models Meet NL2Code: A Survey (2023.acl-long)
Copied to clipboard
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, Jian-Guang Lou
| Challenge: | generating code from a natural language description is a pressing and significant challenge in code intelligence. |
| Approach: | They propose to survey 27 existing large language models for NL2Code and compare them to humanEval benchmarks. |
| Outcome: | The proposed model is compared with existing models on the HumanEval benchmark. |
CoDesc: A Large Code–Description Parallel Dataset (2021.findings-acl)
Copied to clipboard
Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md. Mahim Anjum Haque, Tahmid Hasan, Wasi Ahmad, Anindya Iqbal, Rifat Shahriyar
| Challenge: | Existing models for natural language and programming languages are lagging behind due to a lack of large datasets and benchmarks. |
| Approach: | They present a large parallel dataset of Java methods and natural language descriptions that is used to train deep neural models. |
| Outcome: | The proposed dataset improves code summarization and code search by 22% and opens up possibilities for pretrained language models for Java. |
Exploring Data Augmentation for Code Generation Tasks (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have impacted how models are trained for programming language tasks. |
| Approach: | They propose to use augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% respectively. |
| Outcome: | The proposed methods improve translation and summarization by 6.9% and 7.5% respectively. |