SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
Similar Papers
Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Sentence embedding is essential for many NLP tasks, but reliance on manual labels limits scalability. |
| Approach: | They propose a method for controlling the generation direction of large language models in the latent space by integrating ranking information and semantic information. |
| Outcome: | The proposed method achieves new SOTA performance with a modest cost in ranking sentence synthesis. |
Generating Datasets with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles. |
| Approach: | They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models. |
| Outcome: | The proposed approach outperforms baselines on several semantic textual similarity datasets. |
GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings (2025.findings-naacl)
Copied to clipboard
| Challenge: | Training-free embedding methods focus on optimizing embeddable prompts . previous methods have overlooked the benefits of utilizing generative abilities of LLMs - GenEOL . |
| Approach: | They propose a method that leverages pretrained large language models to embed text . they propose generating diverse transformations of a sentence that preserve its meaning . |
| Outcome: | The proposed method outperforms existing training-free embedding methods by 2.85 points on the sentence semantic text similarity (STS) benchmark. |
Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work uses sentences within the same batch as negatives, which suffers from easy negatives. |
| Approach: | They propose to align sentence representations from different languages into a unified embedding space . they adapt MoCo to further improve the quality of alignment . |
| Outcome: | The proposed model achieves state-of-the-art on several tasks. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Scaling Sentence Embeddings with Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current methods based on contrastive learning have generated high-quality sentence embeddings. |
| Approach: | They propose a method to enhance LLM performance on sentence embeddings with a one-word limitation. |
| Outcome: | The proposed method outperforms contrastive learning methods on sentence embeddings without fine-tuning and with fine-untun. |
Co-Evolving LLMs and Embedding Models via Density-Guided Preference Optimization for Text Clustering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text clustering use static pseudo-oracles, i.e., unidirectionally querying them for similarity assessment or data augmentation. |
| Approach: | They propose a training framework that enables bidirectional refinement between LLMs and embedding models by using task-aware prompts to guide the LLM in generating interpretations for the input texts. |
| Outcome: | Experiments on 14 benchmark datasets across 5 tasks demonstrate the effectiveness of the proposed training framework. |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2023) show that text embeddings from large language models can be aligned with key tokens in input text. |
| Approach: | They propose a sparse retrieval method based on aligned tokens for large language models . they show that this phenomenon is universal and is not affected by model architecture . |
| Outcome: | The proposed method can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |