Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern document retrieval embedding methods typically encode passages (chunks) from documents independently, often overlooking contextual information from the rest of the document. |
| Approach: | They propose a benchmark to evaluate retrieval models' ability to leverage document-wide context. |
| Outcome: | The proposed method significantly improves retrieval quality on ConTEB without sacrificing base model performance. |
Similar Papers
Situated Embedding Models for Context-Aware Dense Retrieval (2026.acl-short)
Copied to clipboard
| Challenge: | Existing embedding models are not well-equipped to encode situated context effectively, i.e., situating a chunk’s meaning within its context. |
| Approach: | They propose to represent short chunks in a way that is conditioned on a broader context window to enhance retrieval performance. |
| Outcome: | The proposed model outperforms state-of-the-art embedding models on a book-plot retrieval dataset. |
CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Contextual document embedding reranking is an efficient and efficient retrieval framework. |
| Approach: | They propose a highly efficient retrieval framework that uses contextual document embedding reranking to incorporate ranking context into training. |
| Outcome: | The proposed framework reduces the computational overhead of a first-stage method and can be used as stand-alone retrieval models. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Contextual embeddings have shown state-of-the-art performance for various tasks such as question answering, sentiment analysis, and textual similarity. |
| Approach: | They propose to integrate transformer-based neural language models into their models to achieve relative improvements of up to 36% on granular tasks. |
| Outcome: | The proposed model outperforms baselines for more granular tasks while outperforming TF-IDF for more complex tasks. |
Diffusion-Pretrained Dense and Contextual Embeddings (2026.acl-industry)
Copied to clipboard
| Challenge: | pplx-embed uses diffusion-based pretraining to capture bidirectional context within passages. |
| Approach: | They propose a family of multilingual embedding models that leverage bidirectional attention through diffusion-based pretraining to capture bidirectional context within passages. |
| Outcome: | The proposed models achieve competitive performance on the MTEB(Multilingual, v2), MTEF(Code), BERGEN, and ToolRet retrieval benchmarks while pplx-embed-context-v1 sets new records on the ConTEB benchmark. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
Pooled Contextualized Embeddings for Named Entity Recognition (N19-1)
Copied to clipboard
| Challenge: | Contextual string embeddings are a recent type of word embeddable that are useful for sequence labeling tasks. |
| Approach: | They propose a method that dynamically aggregates contextualized embeddings of each unique string . they then use a pooling operation to distill a ”global” word representation from all contextualized instances . |
| Outcome: | The proposed method improves state-of-the-art for named entity recognition tasks. |
Leveraging Cognitive Complexity of Texts for Contextualization in Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to estimate semantic similarity of queries and documents rely on token-level information derived from query/document interactions. |
| Approach: | They propose a new DRM that leverages query/document interactions based on full embedding representations generated by a Transformer-based model. |
| Outcome: | The proposed model outperforms fine-tuning techniques on lightweight bi-encoders and traditional late-interaction models. |
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (D19-1)
Copied to clipboard
| Challenge: | Prior work on pretrained sentence embeddings and benchmarks focused on the capabilities of stand-alone sentences. |
| Approach: | They propose a test suite of tasks to evaluate whether sentence representations include broader context information. |
| Outcome: | The proposed training objectives help to encode different aspects of information in document structures. |
Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval augmentation work with chunked contexts, which leads to poor quality of semantic representation and incomplete retrieval of useful information. |
| Approach: | They propose a method for retrieval augmentation of long-context language modeling using landmark embedding. |
| Outcome: | The proposed method outperforms existing retrieval methods with a notable advantage. |