| Challenge: | Existing methods for learning sentence embedding are limited, but still need to be improved. |
| Approach: | They propose a vector-based multi-head attention model that uses special cases of max pooling, mean pooling and scalar self-attention. |
| Outcome: | The proposed model improves on natural language inference, author profiling, and sentiment classification tasks. |
Similar Papers
Sentence Representations via Gaussian Embedding (2024.eacl-short)
Copied to clipboard
| Challenge: | Sentence embeddings represent a sentence's meaning as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations. |
| Approach: | They propose a Gaussian-distribution-based contrastive learning framework for sentence embedding that can handle asymmetric inter-sentential relations and a similarity measure for identifying entailment relations. |
| Outcome: | The proposed framework performs comparable to that of previous methods on natural language inference tasks and estimates direction of entailment relations, which is difficult with point representations. |
CASE – Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent approaches use semantic similarity to improve the quality of sentence embeddings, but it is difficult to measure the similarity between sentences. |
| Approach: | They propose a condition-aware sentence embedding method that uses an LLM encoder to create an embeddable sentence under a given condition. |
| Outcome: | The proposed method improves the performance of LLM-based embeddings and the isotropy of the embeddable space despite requiring a small number of dimensions. |
Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Sentence embedding is essential for many NLP tasks, but reliance on manual labels limits scalability. |
| Approach: | They propose a method for controlling the generation direction of large language models in the latent space by integrating ranking information and semantic information. |
| Outcome: | The proposed method achieves new SOTA performance with a modest cost in ranking sentence synthesis. |
GASE: Generatively Augmented Sentence Encoding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarizing, or extracting keywords, followed by pooling the original and synthetic embeddings. |
| Approach: | They propose a training-free approach to improve sentence embeddings by applying generative text models for data augmentation at inference time. |
| Outcome: | The proposed approach does not require access to model parameters or computational resources typically required for fine-tuning state-of-the-art models. |
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings (2023.emnlp-main)
Copied to clipboard
Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang
| Challenge: | Prior studies diagnose the anisotropy problem in sentence embeddings from pre-trained language models without fine-tuning. |
| Approach: | They propose an unsupervised method that weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. |
| Outcome: | Empirical evaluations show that the proposed method can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks. |
SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. |
| Approach: | They propose a method to deal with dropout noise and a dimension-wise contrastive learning objective to address feature corruption. |
| Outcome: | The proposed method achieves 1.8 points compared to the strong baseline SimCSE and 1.4 points for DiffCSE. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (P18-1)
Copied to clipboard
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, Lawrence Carin
| Challenge: | Existing deep learning architectures to model compositionality in text sequences require a large number of parameters and expensive computations. |
| Approach: | They propose two additional pooling strategies over word embeddings for improved interpretability and hierarchical pooling for spatial (n-gram) information within text sequences. |
| Outcome: | The proposed pooling strategies improve interpretability and preserve spatial (n-gram) information within text sequences. |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
On the Dimensionality of Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on improving the quality of sentence embeddings, but the exploration of sentence dimension is limited. |
| Approach: | They propose a two-step training method where the encoder and pooler are optimized separately to mitigate the overall performance loss in low-dimension scenarios. |
| Outcome: | The proposed method significantly improves the performance of low-dimensional sentence embeddings on seven STS tasks and seven sentence classification tasks. |