BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
Similar Papers
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (D19-1)
Copied to clipboard
| Challenge: | Existing methods for finding similar sentences require multiple inferences . a modern GPU requires 65 hours to find the most similar pair in 10,000 sentences . |
| Approach: | They propose a modification of the pretrained BERT network that uses siamese and triplet networks to derive semantically meaningful sentence embeddings. |
| Outcome: | The proposed method outperforms existing methods on sentence-pair regression tasks. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Understanding why contextualized embeddings work is still an active area of research. |
| Approach: | They propose to use a BERT architecture to encode a sub-word, position and a segment embedding as input representations for each sub- word. |
| Outcome: | The proposed model performs well on single-sentence prediction tasks while swapping segment IDs in paired-sentent tasks. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
PromptBERT: Improving BERT Sentence Embeddings with Prompts (2022.emnlp-main)
Copied to clipboard
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang
| Challenge: | Existing research shows that BERT and RoBERTa are poor in sentence embeddings due to static token embeddable bias and ineffective BERT layers. |
| Approach: | They propose a novel contrastive learning method for better sentence embeddings by using a template denoising technique. |
| Outcome: | The proposed method achieves 2.29 and 2.58 points of improvement compared to SimCSE and RoBERTa in the unsupervised setting. |
SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features (2022.aacl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (S3BERT) embeddings are composed of explainable sub-embeddings that emphasize various sentence meaning features. |
| Approach: | They propose to induce Semantically Structured Sentence BERT embeddings (S3BERT) that emphasize various sentence meaning features. |
| Outcome: | The proposed model shows high correlation to human similarity ratings, but lacks interpretability. |
Representation biases in sentence transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | argued that transformer-based models are not well suited for sentence-level downstream tasks. |
| Approach: | They propose to use sentence transformers to produce full-sentence representations . they propose to combine transformers with a training regime that embeds tokens into the model . |
| Outcome: | The proposed model performs better on downstream tasks than the vanilla model and its variants. |