Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods modify attention mechanism to be bidirectional, undermining LLMs’ ability to extract semantic information acquired during pre-training. |
| Approach: | They propose a general-purpose embedding model that pre-encodes input text into a single Contextual token and then prepends it to the LLM's input sequence. |
| Outcome: | The proposed model improves performance of decoder-only large language models without altering their architectures or introducing significant computational overhead. |
Similar Papers
Tokenizer-Aware Cross-Lingual Adaptation of Decoder-Only LLMs through Embedding Relearning and Swapping (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been primarily focused on English, leaving the multilingual ability unexplored. |
| Approach: | They propose a technique that creates new tokenizers and tunes embeddings on fixed model weights for target language adaptation. |
| Outcome: | The proposed method is light-weight and performant but has limitations for older models and high resource languages. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
Exploring the Hidden Capacity of LLMs for One-Step Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Approach: | They show that large language models can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Outcome: | The proposed model can generate hundreds of accurate tokens in one token-parallel forward pass, when provided with only two learned embeddings. |
KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that decoder-only LLMs can serve as strong embedding backbones when fine-tuned with contrastive objectives. |
| Approach: | They propose a framework that activates the latent representation power of frozen LLMs by rerouting the final token's KV states as a prepended prefix. |
| Outcome: | The proposed framework outperforms existing training-free baselines by 10% on MTEB and maintains robust performance on sequences up to 4,096 tokens. |
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely explored for embedding generation. |
| Approach: | They propose an embedding-based in-context prompt training strategy that leverages in-constext learning to generate high-quality embeddables while reducing computational burden. |
| Outcome: | The proposed method surpasses models trained on publicly available retrieval data and achieves state-of-the-art embedding performance on the MTEB benchmark. |
Causal-LLM: A Unified One-Shot Framework for Prompt- and Data-Driven Causal Graph Discovery (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current causal discovery methods rely on pairwise or iterative strategies that fail to capture global dependencies, amplify local biases, and reduce overall accuracy. |
| Approach: | They propose a framework for one-step full causal graph discovery using prompt-based discovery and a data-driven method for settings without metadata. |
| Outcome: | The proposed framework outperforms state-of-the-art models by approximately 40% in edge accuracy on datasets like Asia and Sachs while maintaining strong performance on more complex graphs. |
Can Post-Training Transform LLMs into Causal Reasoners? (2026.findings-acl)
Copied to clipboard
| Challenge: | Causal inference is a core component of human cognition and requires decision-makers to distinguish between causation and association. |
| Approach: | They propose a dataset comprising seven core causal tasks for training and five diverse test sets and evaluate five different post-training approaches. |
| Outcome: | The proposed model achieves 93.5% accuracy on the CaLM benchmark, compared to 55.4% by OpenAI o3. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Dense retrieval requires discriminative embeddings to represent the semantic relationship between query and document. |
| Approach: | They propose an unsupervised approach that performs unsupervised adaptation of large language models for dense retrieval. |
| Outcome: | The proposed model improves on a variety of dense retrieval benchmarks and is available on github. |