PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation

Challenge:	Pre-trained sequence-to-sequence models are typically pretrained on extensive raw text corpora and fine-tuned on task-specific data.
Approach:	They introduce a pre-trained sequence-to-sequence model trained from scratch for Khmer using carefully curated Khmer and English corpora.
Outcome:	The proposed model outperforms existing models on three generative tasks and is data-efficient and effective in enhancing performance across various natural language generation tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

IndicBART: A Pre-trained Model for Indic Natural Language Generation (2022.findings-acl)

Copied to clipboard

Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, Pratyush Kumar

Text Generation

Challenge:	IndicBART is a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English.
Approach:	They present a multilingual sequence-to-sequence pre-trained model for Indic languages . they evaluate it on two NLG tasks: Neural Machine Translation and extreme summarization .
Outcome:	The proposed model performs well on low-resource translation scenarios . Script sharing, multilingual training, and better utilization contribute to the performance.

Unified Pre-training for Program Understanding and Generation (2021.naacl-main)

Copied to clipboard

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Text Generation

Challenge:	PLUG is a programming language that is used for programming and language understanding and generation tasks.
Approach:	They propose a sequence-to-sequence model that performs a broad spectrum of program and language understanding and generation tasks.
Outcome:	The proposed model outperforms or rivals state-of-the-art models on code summarization, code generation, and code translation tasks in seven programming languages.

FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT (2022.coling-1)

Copied to clipboard

Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Eiichiro Sumita

Multilinguality

Challenge:	FeatureBART is a linguistically motivated sequence-to-sequence monolingual pre-training strategy . syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the pre-trained model .
Approach:	They propose a linguistically motivated sequence-to-sequence monolingual pre-training strategy that incorporates syntactic features into the framework.
Outcome:	The proposed model improves translation quality in bilingual and multilingual settings over models that do not use features.

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)

Copied to clipboard

Sascha Rothe, Shashi Narayan, Aliaksei Severyn

Language Models

Challenge:	Unsupervised pre-training of large neural models has revolutionized Natural Language Processing.
Approach:	They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint.
Outcome:	The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion.

PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)

Copied to clipboard

Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, Zhou Yu

Dialogue Systems & Conversational Agents

Challenge:	Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive.
Approach:	They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks.
Outcome:	The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks.

Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model (2021.findings-acl)

Copied to clipboard

Jeonghyeok Park, Hai Zhao

Language Models

Challenge:	Existing methods to exploit PrLMs for NLG tasks do not get as much performance gain as in the NLU task.
Approach:	They propose a method to integrate public checkpoints of PrLMs for the most convenience.
Outcome:	The proposed method significantly improves the quality of the language generation tasks on 6 different kinds of PrLMs.

HyperT5: Towards Compute-Efficient Korean Language Modeling (2023.acl-industry)

Copied to clipboard

Dongju Park, Soonwon Ka, Kang Min Yoo, Gichang Lee, Jaewook Kang

Language Models

Challenge:	Pretraining and fine-tuning language models is a common practice in NLP, but deploying general-purpose language models without the abundant computation or data resources is proving difficult.
Approach:	They propose a sequence-to-sequence language model architecture that can be more practical and compute-efficient than the decoder-oriented approach.
Outcome:	The proposed language model outperforms competing models in Korean benchmarks and is more efficient in low-resource settings.

mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (2023.findings-emnlp)

Copied to clipboard

Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, Sebastian Ruder

Language Models

Challenge:	Recent large language models display surprising multilingual capabilities despite being pre-trained on English data.
Approach:	They propose a multilingual sequence-to-sequence model that disentangles language-specific information from language-agnostic information.
Outcome:	The proposed model outperforms existing models on representative natural language understanding and generation tasks in 40+ languages.

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)

Copied to clipboard

David Uthus, Santiago Ontanon, Joshua Ainslie, Mandy Guo

Language Models

Challenge:	a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages.
Approach:	They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data .
Outcome:	The proposed model performs well on multilingual summarization and question-answering tasks.

IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation (2021.emnlp-main)

Copied to clipboard

Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, Pascale Fung

Language Models

Challenge:	Lack of publicly available NLG benchmarks for low-resource languages poses a challenge . authors show that IndoBART and IndoGPT achieve competitive performance on all tasks .
Approach:	They propose a benchmark to measure natural language generation progress in three low-resource languages of Indonesia . they use a corpus of pretraining datasets to build their models .
Outcome:	The proposed benchmark measures progress in Indonesian, Javanese, and Sundanese . the results highlight the importance of pretraining on closely related, localized languages .