Adaptation of Deep Bidirectional Transformers for Afrikaans Language (2020.lrec-1)
Copied to clipboard
| Challenge: | Pretrained language models can be trained in unsupervised manner, but can be difficult to implement because of the amount of data and computational resources needed for pretraining. |
| Approach: | They propose a model for Afrikaans based on bidirectional encoder representation from transformers. |
| Outcome: | The proposed model outperforms the existing models in part-of-speech tagging, named-entity recognition, and dependency parsing tasks. |
Similar Papers
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing deep bidirectional language models are limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. |
| Approach: | They propose a deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA) it computes contextual language representations without repetition and shows competitive or even better accuracies than BERT . |
| Outcome: | The proposed model performs six times faster on a reranking task and twelve times faster in a semantic similarity task. |
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) are expensive and limited in inference time . a new benchmark for multi-dialectal Arabic language understanding evaluation is developed . |
| Approach: | They introduce two powerful deep bidirectional transformer-based models, ARBERT and MARBERT . they also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation . |
| Outcome: | The proposed models outperform monolingual models with larger vocabulary and larger datasets in Arabic language understanding evaluation. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating pre-trained models into NMT systems are non-trivial and lack a comparison of the impact that other pre-trainers may have on translation performance. |
| Approach: | They propose to use the input of a bilingual pre-trained language model as the input for NMT encoders and a stochastic layer selection approach to ensure sufficient utilization of contextualized embeddings. |
| Outcome: | The proposed bilingual pre-trained language model outperforms all other pre-train models on the IWSLT’14 dataset and the proposed dual-directional translation model. |
Multilingual Translation via Grafting Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to graft pre-trained (masked) language models to multilingual data are limited, and they lack cross-attention component. |
| Approach: | They propose to graft separately pre-trained (masked) language models for machine translation using monolingual data and parallel data. |
| Outcome: | The proposed method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLUE in en2x directions compared with the multilingual Transformer of the same size. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish (2022.lrec-1)
Copied to clipboard
Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, Anne Goujon
| Challenge: | Pre-trained Language Models such as BERT are ubiquitous in NLP but are scarce for low-resource languages such as Luxembourgish. |
| Approach: | They propose a BERT model for Luxembourgish language that they use to augment pre-training datasets by partially translating text data from a closely related language. |
| Outcome: | The proposed model outperforms the baseline model and the mBERT model in Luxembourgish. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |