Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Understanding why contextualized embeddings work is still an active area of research. |
| Approach: | They propose to use a BERT architecture to encode a sub-word, position and a segment embedding as input representations for each sub- word. |
| Outcome: | The proposed model performs well on single-sentence prediction tasks while swapping segment IDs in paired-sentent tasks. |
Similar Papers
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to improve word embeddings for rare words are limited to uncontextualized word embeds. |
| Approach: | They propose a powerful architecture that can infer high-quality embeddings for rare words . they use the surface form and contexts of a word to interact in a deep architecture . |
| Outcome: | The proposed architecture can infer high-quality embeddings for rare words that are suitable as input representations for deep language models. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes . |
| Approach: | They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set . |
| Outcome: | The results show that static embeddings perform better than PLMs when restricted to a candidate set . |
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (D19-1)
Copied to clipboard
| Challenge: | Existing methods for finding similar sentences require multiple inferences . a modern GPU requires 65 hours to find the most similar pair in 10,000 sentences . |
| Approach: | They propose a modification of the pretrained BERT network that uses siamese and triplet networks to derive semantically meaningful sentence embeddings. |
| Outcome: | The proposed method outperforms existing methods on sentence-pair regression tasks. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
Explaining and Improving BERT Performance on Lexical Semantic Change Detection (2021.eacl-srw)
Copied to clipboard
| Challenge: | Lexical semantic change detection is still a challenging field due to the success of type-based embeddings in SemEval-2020 Task 1 and other NLP tasks. |
| Approach: | They compare the performance of BERT embeddings with results from the word sense disambiguation dataset underlying SemEval-2020 Task 1 and the Italian follow-up task DIACR-Ita. |
| Outcome: | The proposed model outperforms token-based embeddings on lexical semantic change detection tasks. |