| Challenge: | Existing pre-trained language models learn contextualized representations by using unlabeled text data and obtain state of the art results on a multitude of NLP tasks. |
| Approach: | They propose a pre-trained BERT model for Romanian language processing and compare it with multi-lingual models on seven Romanian specific NLP tasks. |
| Outcome: | The proposed model outperforms multi-lingual models on seven Romanian specific NLP tasks on sentiment analysis, dialect and cross-dialect topic identification, and diacritics restoration. |
Similar Papers
The birth of Romanian BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are available in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. |
| Approach: | They propose to use a Romanian transformer-based language model to pretrained a large text corpus to evaluate the model. |
| Outcome: | The proposed model is open-source and can be used in production. |
Distilling the Knowledge of Romanian BERTs Using Multiple Teachers (2022.lrec-1)
Copied to clipboard
Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pais, Dan Tufis
| Challenge: | Existing approaches to train pre-trained language models focus on the English language, thus widening the gap when considering low-resource languages. |
| Approach: | They propose three versions of distilled BERT models for the Romanian language . they argue that the models offer performance comparable to their teachers . |
| Outcome: | The proposed models perform comparable to their teachers, while being twice as fast on a GPU and 35% smaller. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
BERTGen: Multi-task Generation through BERT (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU). |
| Approach: | They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks. |
| Outcome: | The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
On the use of BERT for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on the performance of BERT for implicit discourse relation classification have not been conducted. |
| Approach: | They propose to apply BERT to implicit discourse relation classification by performing additional pre-training on text tailored to discourse relations. |
| Outcome: | The proposed methods outperform previous state-of-the-art models in many tasks. |
RoBERTuito: a pre-trained language model for social media text in Spanish (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models have been used in many natural language processing tasks . some domain-specific models have shown to improve performance in some domains . however, for languages other than English, such models are not widely available . |
| Approach: | They present a pre-trained language model for user-generated text in Spanish . it is based on 500 million tweets and has some cross-lingual abilities . |
| Outcome: | The model outperforms models trained on over 500 million tweets on a benchmark in spanish and english. |