Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Despite recent advances in transformer-based sentence encoders, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations. |
| Approach: | They propose to combine a self-contrastive siamese network and a convex neural Bregman divergence network to train longfomer-based document encoders using an unsupervised contrastive learning method. |
| Outcome: | The proposed model outperforms baseline models on three long document topic classification tasks from the legal and biomedical domains. |
Similar Papers
Self-Discriminative Learning for Unsupervised Document Embedding (N19-1)
Copied to clipboard
| Challenge: | Existing methods for document embedding learning do not consider inter-document relationships. |
| Approach: | They propose to exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective. |
| Outcome: | The proposed method has errors that are 5 to 13% lower than state-of-the-art models and is even more pronounced in scarce label setting. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Sentence embeddings are an important component of many natural language processing systems. |
| Approach: | They propose a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. |
| Outcome: | The proposed approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. |
Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations (2024.naacl-long)
Copied to clipboard
Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, Dong Yu
| Challenge: | Sentence embeddings are typically learned to recognize the semantic relation between two text inputs. |
| Approach: | They introduce a contrastively-learned contextual embedding model for fine-grained semantic representation of text. |
| Outcome: | The proposed model is able to produce contextual embeddings corresponding to different atomic propositions, i.e. semantic equivalence between propositions across different text sequences. |
Self-Guided Contrastive Learning for BERT Sentence Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings from pre-trained Transformers are unclear . a self-guided training method is used to fine-tune BERT in a supervised fashion . |
| Approach: | They propose a contrastive learning method that utilizes self-guidance to improve BERT sentence representations. |
| Outcome: | The proposed method is more effective than baselines on diverse sentence-related tasks and robust to domain shifts. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | a new method for learning unsupervised sentence embeddings is proposed . unsup-SimCSE is biased because of the length information encoded into the sentence embeds . |
| Approach: | They propose a new unsupervised sentence embedding method that uses dropout to obtain positive pairs from a pre-trained Transformer encoder. |
| Outcome: | The proposed method outperforms the state-of-the-art unsup-SimCSE on a STS task. |
Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to model long documents are difficult due to the quadratic complexity of text length. |
| Approach: | They propose a hierarchical interactive Transformer for efficient long document modeling. |
| Outcome: | Extensive experiments on three benchmark datasets validate the efficiency and effectiveness of Hi-Transformer in long document modeling. |
RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing PLMs suffer from poor robustness in adversarial scenarios, despite their success with unseen samples. |
| Approach: | They propose a self-supervised sentence embedding framework that enhances generalization and robustness in various text representation tasks and against diverse adversarial attacks. |
| Outcome: | The proposed framework improves generalization and robustness in various representation tasks and against diverse adversarial attacks. |
Composition-contrastive Learning for Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows potential to learn vector representations from unlabelled data without task-specific fine-tuning. |
| Approach: | They propose to maximize alignment between textual embeddings and a composition of their phrasal constituents. |
| Outcome: | The proposed approach improves on similarity tasks comparable to state-of-the-art approaches. |