PyMT5: multi-mode translation of natural language and Python code with transformers

Challenge:	Using Python method text-to-text transfer transformers, developers can easily model source code and natural language.
Approach:	They propose a Python method text-to-text transfer transformer that can translate between all pairs of Python method feature combinations.
Outcome:	The proposed model outperforms similar-sized auto-regressive language models on a large-scale parallel corpus of 26 million methods and 7.7 million method-docstring pairs on the CodeSearchNet test set.

Similar Papers

Overview

Challenge

Approach

Outcome

Unified Pre-training for Program Understanding and Generation (2021.naacl-main)

Copied to clipboard

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Text Generation

Challenge:	PLUG is a programming language that is used for programming and language understanding and generation tasks.
Approach:	They propose a sequence-to-sequence model that performs a broad spectrum of program and language understanding and generation tasks.
Outcome:	The proposed model outperforms or rivals state-of-the-art models on code summarization, code generation, and code translation tasks in seven programming languages.

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021.emnlp-main)

Copied to clipboard

Yue Wang, Weishi Wang, Shafiq Joty, Steven C.H. Hoi

Language Models

Challenge:	Pre-trained models for Natural Languages (NL) like BERT and GPT have been shown to transfer well to Programming Languages.
Approach:	They propose a unified pre-trained encoder-decoder Transformer model that leverages the code semantics conveyed from the developer-assigned identifiers.
Outcome:	The proposed model outperforms existing models on understanding and generation tasks and can capture semantic information from code.

Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)

Copied to clipboard

Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, Thien Huu Nguyen

Syntactic Text Processing

Challenge:	Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing.
Approach:	They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages.
Outcome:	The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks.

AraT5: Text-to-Text Transformers for Arabic Language Generation (2022.acl-long)

Copied to clipboard

El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed

Language Models

Challenge:	Existing models that convert text-based language problems into text-to-text format are not suitable for multilingual tasks.
Approach:	They propose a unified Transformer framework that converts all language problems into a text-to-text format.
Outcome:	The proposed model performs better on all ARGEN tasks than existing models with 49 less data.

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)

Copied to clipboard

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel

Text Generation

Challenge:	Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English.
Approach:	They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages.
Outcome:	The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting.

CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)

Copied to clipboard

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, Steven Hoi

Language Models

Challenge:	Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task.
Approach:	They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions.
Outcome:	The proposed model outperforms open-source LLMs on 20 code-related benchmarks.

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)

Copied to clipboard

David Uthus, Santiago Ontanon, Joshua Ainslie, Mandy Guo

Language Models

Challenge:	a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages.
Approach:	They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data .
Outcome:	The proposed model performs well on multilingual summarization and question-answering tasks.

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)

Copied to clipboard

Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, Chris Dyer

Language Models

Challenge:	a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions.
Approach:	They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions.
Outcome:	The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics.

How Large Language Models are Transforming Machine-Paraphrase Plagiarism (2022.emnlp-main)

Copied to clipboard

Jan Philip Wahle, Terry Ruas, Frederic Kirstein, Bela Gipp

Text Generation

Challenge:	Autoregressive paraphrasing tools can be used to generate convincing plagiarized texts with minimal effort.
Approach:	They evaluate the detection performance of large autoregressive models for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia.
Outcome:	The proposed models generate paraphrases indistinguishable from original work and human experts rate the quality of generated examples as high as originals.

Multi-Hop Transformer for Document-Level Machine Translation (2021.naacl-main)

Copied to clipboard

Long Zhang, Tong Zhang, Haibo Zhang, Baosong Yang, Wei Ye, Shikun Zhang

Machine Translation

Challenge:	Existing approaches to document-level neural machine translation (NMT) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process.
Approach:	They propose a novel multi-hop Transformer which explicitly models the human-like draft-editing and reasoning process by attending to multiple antecedent sentences iteratively.
Outcome:	Experiments on four widely used document translation tasks show that the proposed model significantly improves document-level translation performance and tackles discourse phenomena such as coreference error and the problem of polysemy.