| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
Similar Papers
BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features (2022.aacl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (S3BERT) embeddings are composed of explainable sub-embeddings that emphasize various sentence meaning features. |
| Approach: | They propose to induce Semantically Structured Sentence BERT embeddings (S3BERT) that emphasize various sentence meaning features. |
| Outcome: | The proposed model shows high correlation to human similarity ratings, but lacks interpretability. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
Improving Contextual Representation with Gloss Regularized Pre-training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that the gloss regularizer module enhances word semantic similarity in pre-training. |
| Approach: | They propose an auxiliary gloss regularizer module to BERT pre-training to enhance word semantic similarity. |
| Outcome: | The proposed model improves word similarity in word-level and sentence-level representation. |
Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings (2022.lrec-1)
Copied to clipboard
| Challenge: | Understanding why contextualized embeddings work is still an active area of research. |
| Approach: | They propose to use a BERT architecture to encode a sub-word, position and a segment embedding as input representations for each sub- word. |
| Outcome: | The proposed model performs well on single-sentence prediction tasks while swapping segment IDs in paired-sentent tasks. |
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent pre-trained language models such as BERT have led to noticeable improvements in semantic similarity detection. |
| Approach: | They propose to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. |
| Outcome: | The proposed method improves on multiple semantic similarity datasets and shows that it is beneficial and currently missing from the original model. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (D19-1)
Copied to clipboard
| Challenge: | Existing methods for finding similar sentences require multiple inferences . a modern GPU requires 65 hours to find the most similar pair in 10,000 sentences . |
| Approach: | They propose a modification of the pretrained BERT network that uses siamese and triplet networks to derive semantically meaningful sentence embeddings. |
| Outcome: | The proposed method outperforms existing methods on sentence-pair regression tasks. |
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)
Copied to clipboard
| Challenge: | Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives. |
| Approach: | They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining. |
| Outcome: | The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks. |
More Discriminative Sentence Embeddings via Semantic Graph Smoothing (2024.eacl-short)
Copied to clipboard
| Challenge: | Text categorization is a natural language processing task that involves arranging texts into coherent groups based on their content. |
| Approach: | They propose to use semantic graph smoothing to enhance sentence embeddings from pretrained models to improve results for supervised and unsupervised document categorization tasks. |
| Outcome: | The proposed method improves sentences embeddings for supervised and unsupervised document categorization tasks. |