| Challenge: | pplx-embed uses diffusion-based pretraining to capture bidirectional context within passages. |
| Approach: | They propose a family of multilingual embedding models that leverage bidirectional attention through diffusion-based pretraining to capture bidirectional context within passages. |
| Outcome: | The proposed models achieve competitive performance on the MTEB(Multilingual, v2), MTEF(Code), BERGEN, and ToolRet retrieval benchmarks while pplx-embed-context-v1 sets new records on the ConTEB benchmark. |
Similar Papers
Query-as-context Pre-training for Dense Passage Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve passage retrieval performance by using context-supervised pre-training are weakly correlated. |
| Approach: | They propose to use query-as-context pre-training to train passage-query pairs . they evaluate the pre-trained models on large-scale passage retrieval benchmarks . |
| Outcome: | The proposed technique improves performance on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern document retrieval embedding methods typically encode passages (chunks) from documents independently, often overlooking contextual information from the rest of the document. |
| Approach: | They propose a benchmark to evaluate retrieval models' ability to leverage document-wide context. |
| Outcome: | The proposed method significantly improves retrieval quality on ConTEB without sacrificing base model performance. |
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval. |
| Approach: | They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning. |
| Outcome: | The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect. |
Contextualized Query Embeddings for Conversational Search (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to conversational search use multiple inference pipelines that require long inference times . despite their effectiveness, such a pipeline often includes multiple neural models that require longer inference time. |
| Approach: | They propose to integrate conversational query reformulation directly into a dense retrieval model . they use a dataset with pseudo-relevance labels to overcome the lack of training data . |
| Outcome: | The proposed model rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that multilingual generative models exhibit a strong language bias toward high-resource languages. |
| Approach: | They propose a cross-lingual alignment framework exploiting pairs of translation sentences to improve cross-linguistic abilities. |
| Outcome: | The proposed framework improves cross-lingual abilities and mitigates performance gap. |
Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning. |
| Approach: | They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations. |
| Outcome: | The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs. |
Diverse Pretrained Context Encodings Improve Document Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for sentence-level sequence-to-sequence translations do not use extra-sentential information. |
| Approach: | They propose a sentence-level sequence-to-sequence transformer with multiple pre-trained context signals. |
| Outcome: | The proposed model outperforms existing models on Chinese-English and English-German tasks. |