ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing models for the English language have been used to train on large corpus of high-quality texts. |
| Approach: | They present a pretrained Transformer-based encoder-decoder model for the Vietnamese language . they benchmark ViT5 on two downstream text generation tasks . |
| Outcome: | The proposed model outperforms existing models on Vietnamese Abstractive Summarization and Named Entity Recognition tasks. |
Similar Papers
ViDeBERTa: A powerful pre-trained language model for Vietnamese (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing models for Vietnamese that perform well on downstream tasks, such as Question answering, are based on Transformer. |
| Approach: | They propose a pre-trained monolingual Vietnamese model with three versions . they fine-tune and evaluate the model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. |
| Outcome: | The proposed model outperforms the existing model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. |
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2022.acl-long)
Copied to clipboard
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
| Challenge: | Existing work shows that pre-trained models can improve in various natural language processing tasks. |
| Approach: | They propose a unified-modal encoder-decoder framework that pre-trains speech-text representations using large-scale unlabeled speech and text data. |
| Outcome: | The proposed framework is superior to existing models on speech-to-text processing tasks. |
ViHateT5: Enhancing Hate Speech Detection in Vietnamese With a Unified Text-to-Text Transformer Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for HSD in Vietnamese focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia. |
| Approach: | They propose a T5-based HSD model pre-trained on a domain-specific dataset . their results highlight the significance of label distribution in pre-training data on model efficacy. |
| Outcome: | The proposed model can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. |
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks. |
| Approach: | They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection . |
| Outcome: | The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters. |
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages. |
| Approach: | They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data . |
| Outcome: | The proposed model performs well on multilingual summarization and question-answering tasks. |
AraT5: Text-to-Text Transformers for Arabic Language Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models that convert text-based language problems into text-to-text format are not suitable for multilingual tasks. |
| Approach: | They propose a unified Transformer framework that converts all language problems into a text-to-text format. |
| Outcome: | The proposed model performs better on all ARGEN tasks than existing models with 49 less data. |
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021.emnlp-main)
Copied to clipboard
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, Furu Wei
| Challenge: | Multilingual T5 pretrains a sequence-to-sequence model on monolingual texts, but it has shown promising results on many cross-lingual tasks. |
| Approach: | They propose a partially non-autoregressive objective for text-to-text pre-training and propose mT6 to improve cross-lingual transferability over multilingual T5. |
| Outcome: | The proposed model improves cross-lingual transferability over existing models. |
PhoBERT: Pre-trained language models for Vietnamese (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that PhoBERT outperforms the recent best pre-trained multilingual model XLM-R in multiple Vietnamese-specific NLP tasks. |
| Approach: | They present PhoBERT with two versions, Phobert-base and PhoBRET-large, which are pre-trained for Vietnamese. |
| Outcome: | The proposed model outperforms the best pre-trained model XLM-R and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. |
UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation (2023.acl-short)
Copied to clipboard
| Challenge: | Existing text-based recommendation frameworks that use pretrained language models (PLMs) can improve performance on text-related tasks. |
| Approach: | They propose a unified local- and global-attention Transformer encoder to better model two-level contexts of user history. |
| Outcome: | The proposed framework improves on three text-based recommendation tasks. |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)
Copied to clipboard
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
| Challenge: | Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English. |
| Approach: | They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. |
| Outcome: | The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting. |