Longtriever: a Pre-trained Long Text Encoder for Dense Document Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing PLMs are infeasible for processing long documents due to computational costs and incomprehensive document understanding. |
| Approach: | They propose a retrieval model that models local semantics and global context semantics in a tightly-coupled manner. |
| Outcome: | The proposed model overcomes three core challenges of long document retrieval: substantial computational cost, incomprehensive document understanding, and scarce annotations. |
Similar Papers
Domain-matched Pre-training Tasks for Dense Retrieval (2022.findings-naacl)
Copied to clipboard
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, Yashar Mehdad
| Challenge: | Existing approaches to improve performance of pre-training tasks are needed. |
| Approach: | They propose to pre-train large bi-encoder models on a recently released set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting reddit conversation dataset. |
| Outcome: | The proposed model can be pre-trained on a set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting dataset of Reddit conversations. |
Condenser: a Pre-training Architecture for Dense Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work fine-tunes deep LMs to encode text sequences into single dense vector representations, but dense encoders require a lot of data and sophisticated techniques to train and suffer in low data situations. |
| Approach: | They propose to pre-train Transformer language models (LMs) with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. |
| Outcome: | The proposed model improves on various text retrieval and similarity tasks by large margins over standard LMs. |
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have limited generalization capabilities and performance challenges. |
| Approach: | They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency . |
| Outcome: | The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency. |
mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024.emnlp-industry)
Copied to clipboard
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, Min Zhang
| Challenge: | Existing models for text retrieval are based on a multi-stage process that involves retrieving documents from a large corpus. |
| Approach: | They propose to build a multilingual text representation model and a cross-encoder reranker from scratch for text retrieval. |
| Outcome: | The proposed models outperform the state-of-the-art models on long-context retrieval benchmarks. |
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Considerable efforts have been and are still being put into increasing the context length of Large Language Models (LLMs) |
| Approach: | They propose an approach that divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. |
| Outcome: | The proposed approach outperforms 8 state-of-the-art methods in effectiveness and efficiency for document summarization and question answering, and achieves the best performance on LongBench v2 among models of comparable size. |
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder (2021.emnlp-main)
Copied to clipboard
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, Arnold Overwijk
| Challenge: | Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space. |
| Approach: | They propose a self-learning method that pre-trains the autoencoder using a weak decoder to push the encoder to provide better sequence representations. |
| Outcome: | The proposed model significantly boosts the effectiveness and few-shot ability of dense retrieval models on web search, news recommendation, and open domain question answering. |
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-cuning techniques. |
| Approach: | They propose a pre-training architecture that learns to condense information into the dense vector through LM pre-training and a coCondenser architecture which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. |
| Outcome: | The proposed architecture reduces the need for heavy data engineering and large batch training. |
Efficient Long-Text Understanding with Short-Text Models (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing transformer-based pretrained language models cannot be applied to long sequences due to their quadratic complexity. |
| Approach: | They propose a simple approach to long sequences that re-uses battle-tested short-text pretrained LMs. |
| Outcome: | The proposed approach is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. |
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Dense retrieval requires discriminative embeddings to represent the semantic relationship between query and document. |
| Approach: | They propose an unsupervised approach that performs unsupervised adaptation of large language models for dense retrieval. |
| Outcome: | The proposed model improves on a variety of dense retrieval benchmarks and is available on github. |
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Dense retrievers have impressive performance, but their demand for abundant training data limits their application scenarios. |
| Approach: | They propose a method which uses unlabeled data to construct pseudo-positive examples from unlabelled data and then contrastively weighs the contrastive loss of different pairs according to the estimated relevance. |
| Outcome: | The proposed method beats the SOTA unsupervised Contriever model on BEIR and open-domain QA retrieval benchmarks and is a good few-shot learner. |