Partially-Random Initialization: A Smoking Gun for Binarization Hypothesis of BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained BERT has been used for natural language processing tasks but its performance is limited by memory and computational complexity. |
| Approach: | They propose to use pre-trained BERT to achieve decent accuracy . they propose to combine binary BERT with a randomly-initialized encoder . |
| Outcome: | The proposed model achieves state-of-the-art on GLUE and SQuAD benchmarks. |
Similar Papers
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Active Learning for BERT: An Empirical Study (2020.emnlp-main)
Copied to clipboard
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim
| Challenge: | Existing approaches to deal with data scarcity are active learning (AL) and pre-trained models are not being considered. |
| Approach: | They propose to use active learning techniques to cope with data scarcity in binary text classification scenarios where the annotation budget is very small and the data is often skewed. |
| Outcome: | The proposed methods improve BERT performance in binary text classification scenarios where the annotation budget is very small and the data is often skewed. |
BinaryBERT: Pushing the Limit of BERT Quantization (2021.acl-long)
Copied to clipboard
| Challenge: | Recent pre-trained language models have achieved remarkable performance improvement in various tasks, but the improvement generally comes at the cost of increasing model size and computation. |
| Approach: | They propose a binary quantization technique which initializes binaryBERT by splitting from a ternary network. |
| Outcome: | The proposed model achieves state-of-the-art performance on the GLUE and SQUAD benchmarks while being 24x smaller. |
Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evidence suggests that pre-trained Transformers encode commonsense knowledge . however, the extent to which this knowledge is acquired is unclear . |
| Approach: | They inject verbalized knowledge into pre-training minibatches and evaluate generalization . they find generalization does not improve over the course of pre- training from scratch . |
| Outcome: | The proposed model generalizes to supported inferences after pre-training on the injected knowledge. |
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work has explored the suitability of pre-trained language models in low resource settings with less than 1,000 training data points. |
| Approach: | They propose to use pool-based active learning to speed up training while keeping the cost of labeling new data constant. |
| Outcome: | The proposed model can be fine-tuned to optimize for low-resource settings while keeping the cost of labeling constant. |
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent years have witnessed the widespread use of transfer learning techniques in Natural Language Processing (NLP) |
| Approach: | They train BERT models from scratch using many configurations involving general and medical corpora. |
| Outcome: | The initial corpus only has a weak influence when these are further pre-trained on a medical corpus. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |
How Far Is Too Far? Studying the Effects of Domain Discrepancy on Masked Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform strongly on a wide variety of NLP tasks. |
| Approach: | They propose a mechanism to quantify the difference in domains between the pre-trained model and the task and partition it using a cloze task. |
| Outcome: | The proposed model performs better on openly available e-commerce datasets than the original model on scientific and biomedical datasets. |
How to Train BERT with an Academic Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models such as BERT are used in many NLP tasks, but their pretraining phase can be prohibitively expensive for startups and academic research groups. |
| Approach: | They propose a recipe for pretraining a large language model in 24 hours using a low-end deep learning server. |
| Outcome: | The proposed model can be trained on GLUE tasks at fraction of the cost of pretraining. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |