A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2023) show that text embeddings from large language models can be aligned with key tokens in input text. |
| Approach: | They propose a sparse retrieval method based on aligned tokens for large language models . they show that this phenomenon is universal and is not affected by model architecture . |
| Outcome: | The proposed method can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly. |
Similar Papers
Enhancing Lexicon-Based Text Embeddings with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. |
| Approach: | They introduce the first lexicon-based embeddings that consolidates the vocabulary space through token embeddation clustering to handle the issue of token redundancy in LLM vocabularies. |
| Outcome: | The proposed model outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) it also supports efficient dimension pruning without any specialized objectives like Matryoshka Representation Learning. |
Exploring the Hidden Capacity of LLMs for One-Step Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Approach: | They show that large language models can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Outcome: | The proposed model can generate hundreds of accurate tokens in one token-parallel forward pass, when provided with only two learned embeddings. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
Training LLMs to be Better Text Embedders through Bidirectional Reconstruction (2025.emnlp-main)
Copied to clipboard
Chang Su, Dengliang Shi, Siyuan Huang, Jintao Du, Changhua Meng, Yu Cheng, Weiqiang Wang, Zhouhan Lin
| Challenge: | Existing text embedding approaches often leverage the embeddment of the final token, typically a reserved special token such as ‘[EOS]‘. |
| Approach: | They propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. |
| Outcome: | The proposed training stage improves performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
Aligners: Decoupling LLMs and Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. |
| Approach: | They propose to decouple LLMs and alignment by training *aligner* models that can be used to align any LLM on an as-needed basis. |
| Outcome: | The proposed model can be used to align any LLM for a given criteria on an as-needed basis. |
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |
KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that decoder-only LLMs can serve as strong embedding backbones when fine-tuned with contrastive objectives. |
| Approach: | They propose a framework that activates the latent representation power of frozen LLMs by rerouting the final token's KV states as a prepended prefix. |
| Outcome: | The proposed framework outperforms existing training-free baselines by 10% on MTEB and maintains robust performance on sequences up to 4,096 tokens. |
Constructing Your Model’s Value Distinction: Towards LLM Alignment with Anchor Words Tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models (LLMs) shows that they can generate outputs that are honest, positive, harmless, etc. |
| Approach: | They propose a method that amplifies logits difference between positive and negative tokens . they propose to use the logits gap to generate positive and positive tokens after alignment . |
| Outcome: | The proposed method achieves effective alignment, but requires fewer computational resources compared to training-time alignment methods. |