| Challenge: | Spanish language models have demonstrated proficiency in natural language understanding and generation, but there is a scarcity of encoder-decoder models specifically designed for sequence-to-sequence tasks. |
| Approach: | They propose to implement encoder-decoder architectures pre-trained on Spanish corpora . they use them to assess sequence-to-sequence tasks including summarization, question answering . |
| Outcome: | The proposed models outperform models on sequence-to-sequence tasks in Spanish . the models show that they perform well across all tasks, the authors note . |
Similar Papers
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
GreekBART: The First Pretrained Greek Sequence-to-Sequence Model (2024.lrec-main)
Copied to clipboard
Iakovos Evdaimon, Hadi Abdine, Christos Xypolopoulos, Stamatis Outsios, Michalis Vazirgiannis, Giorgos Stamou
| Challenge: | Transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing. |
| Approach: | They introduce a new language model, GreekBART, that is based on a BART-base architecture. |
| Outcome: | The proposed model outperforms BERT, GPT and other transformer-based models on discriminative tasks. |
BARThez: a Skilled Pretrained French Sequence-to-Sequence Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Inductive transfer learning has taken the entire NLU field by storm, with models such as BERT and BART setting new state-of-the-art on countless tasks. |
| Approach: | They introduce a large-scale pretrained seq2seq model for French that is very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. |
| Outcome: | The proposed model outperforms existing models on discriminative and generative tasks on a French summarization dataset. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained encoder-only and sequence-to-sequence models are computationally expensive. |
| Approach: | They propose a recipe to initialize one model from the other to improve pre-training efficiency. |
| Outcome: | The proposed method matches the performance of a from-scratch model with a multilingual encoder while reducing the total compute cost by 27%. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
Multilingual Sequence-to-Sequence Models for Hebrew NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on pretrained language models for Hebrew is under-parameterized and under-trained . previous work on pretraining Hebrew LMs focused on encoder-only architectures . |
| Approach: | They propose to use sequence-to-sequence generative architectures to train large LMs in morphologically rich languages such as Hebrew. |
| Outcome: | The proposed model improves on all existing Hebrew NLP benchmarks. |
Improving AMR Parsing with Sequence-to-Sequence Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) parsing is limited by the size of curated datasets. |
| Approach: | They propose a seq2seq pre-training approach to build pre-trained models on three relevant tasks. |
| Outcome: | The proposed model improves performance on three relevant tasks while maintaining the response of pre-trained models. |
Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence-to-Sequence (S2S) models have been successful on text generation tasks . however, learning complex structures with S2S models remains challenging . |
| Approach: | They propose to use constrained decoding to model part-of-speech tagging, named entity recognition, constituency, and dependency parsing tasks with 3 lexically diverse linearization schemas and corresponding constrained coding methods. |
| Outcome: | The proposed methods outperform the state-of-the-art on four core tasks. |