EigenSent: Spectral sentence embeddings using higher-order Dynamic Mode Decomposition (P19-1)
Copied to clipboard
| Challenge: | Existing methods for calculating word embeddings use weighted averages or learn word representations, but this is not the case. |
| Approach: | They propose to use spectral methods of signal representation and summarization to construct word-sequence embeddings in unsupervised fashion by using a spectrally-based decomposition and signal summarizing technique on text. |
| Outcome: | The proposed method can summarize transitions in a sequence of words and generate an embedding that can represent well the sequence itself. |
Similar Papers
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
Sentence Representations via Gaussian Embedding (2024.eacl-short)
Copied to clipboard
| Challenge: | Sentence embeddings represent a sentence's meaning as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations. |
| Approach: | They propose a Gaussian-distribution-based contrastive learning framework for sentence embedding that can handle asymmetric inter-sentential relations and a similarity measure for identifying entailment relations. |
| Outcome: | The proposed framework performs comparable to that of previous methods on natural language inference tasks and estimates direction of entailment relations, which is difficult with point representations. |
Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings do not capture the semantic similarity of sentences. |
| Approach: | They propose a framework that integrates compositional sentence operations into the embedding space and optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddables. |
| Outcome: | The proposed framework improves the interpretability of sentence embeddings on four textual generation tasks while maintaining strong performance on traditional semantic similarity tasks. |
WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on learning sentence semantics when fine-tuned with supervised data. |
| Approach: | They conduct a thorough examination of pretrained model based unsupervised sentence embeddings. |
| Outcome: | The proposed approach improves on whitening-based vector normalization with less than 10 lines of code. |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing word embeddings combine complementary strengths of their components to achieve unsupervised semantic similarity (STS). |
| Approach: | They propose to ensemble pre-trained sentence encoders into sentence meta-embeddings to achieve unsupervised Semantic Textual Similarity (STS) they adapt dimensionality reduction, generalized Canonical Correlation Analysis and cross-view auto-encoders to their work. |
| Outcome: | The proposed method achieves 3.7% to 6.4% Pearson’s r over single-source word embeddings on the STS Benchmark and on the StS12-STS16 datasets. |
DefSent: Sentence Embeddings using Definition Sentences (2021.acl-short)
Copied to clipboard
| Challenge: | Sentence embedding methods using natural language inference datasets are limited for limited languages due to large datasets. |
| Approach: | They propose a sentence embedding method that uses definition sentences from a word dictionary. |
| Outcome: | The proposed method performs comparably on unsupervised semantics textual similarity tasks and slightly better on SentEval tasks than methods using large NLI datasets. |