| Challenge: | a small, specialized corpus can produce effective representations, but the quality of pre-training is not affected by the choice of corpus. |
| Approach: | They focus on the representation quality achieved through pre-training alone . |
| Outcome: | The results show that pre-training on a small, specialized corpus produces effective representations. |
Similar Papers
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained Transformers has focused on learning the meaning of positions . Embedding the position information in the self-attention mechanism is also an indispensable factor in NLP . |
| Approach: | They propose to use feature-level analysis to examine pre-trained Transformers' position embeddings . they also use empirical experiments to determine the appropriate positional encoding function . |
| Outcome: | The results of the empirical study can guide future work to choose the appropriate positional encoding function for specific tasks. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent years have witnessed the widespread use of transfer learning techniques in Natural Language Processing (NLP) |
| Approach: | They train BERT models from scratch using many configurations involving general and medical corpora. |
| Outcome: | The initial corpus only has a weak influence when these are further pre-trained on a medical corpus. |
Exploring Quantization for Efficient Pre-Training of Transformer Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization has proven to be effective after pre-training and during fine-tuning, but its effects on pre-trainer performance have remained unexplored. |
| Approach: | They propose a linear quantization strategy to be applied during the pre-training of Transformers to improve model efficiency and stability. |
| Outcome: | The proposed method improves model efficiency, stability, and performance while maintaining language modeling ability. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Are Pretrained Convolutions Better than Pretrained Transformers? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent research has shown promise in entirely convolutional, or CNN, architectures, but they have not been explored using the pre-train-fine-tune paradigm. |
| Approach: | They propose to use the pre-train-fine-tune paradigm to study convolutional models. |
| Outcome: | The proposed architectures outperform Transformers in certain scenarios, but with caveats. |
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Task-agnostic data augmentations have proven widely effective in computer vision, even on pretrained models. |
| Approach: | They examine the effects of two types of task-agnostic data augmentation on pretrained transformers using 5 classification tasks and 6 datasets. |
| Outcome: | The proposed techniques improve performance on 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |