Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to information retrieval (IR) and natural language processing (NLP) use contextual language models, which can improve both synonymy and polysemy problems associated with words. |
| Approach: | They propose an ultra-high dimensional representation scheme equipped with directly controllable sparsity and a bucketing method where embeddings from multiple layers of BERT are selected/merged to represent diverse linguistic aspects. |
| Outcome: | The proposed representation scheme outperforms sparse models with MS MARCO and TREC CAR, and shows that it is highly efficient for storage and search. |
Similar Papers
Sparse, Dense, and Attentional Representations for Text Retrieval (2021.tacl-1)
Copied to clipboard
| Challenge: | Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. |
| Approach: | They propose a dual-encoder-based neural model that combines the efficiency of dual encoders with expressiveness of more costly attentional architectures. |
| Outcome: | The proposed model outperforms strong alternatives in large-scale retrieval. |
The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes (2021.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that dense representations outperform sparse representations with large index sizes. |
| Approach: | They propose to use dense low-dimensional representations to retrieve relevant documents . they show performance decreases quicker for increasing index sizes than for sparse representations . |
| Outcome: | The proposed representations outperform sparse representations with large index sizes. |
LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing text embeddings with high dimensions are difficult to trace and interpret. |
| Approach: | They propose low-dimensional and interpretable text embeddings with relative representations that encode semantic meanings in a vector space where similar texts are close together in the representation space. |
| Outcome: | The proposed embeddings outperform existing models on multiple tasks with fewer dimensions and are lowdimensional and dense while maintaining interpretability. |
Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods suffer from a lack of interpretability . we propose a new interpretability framework that decomposes dense embeddings into distinct, interpretable latent concepts. |
| Approach: | They propose a new interpretability framework that leveragesSparse Autoencoders to decompose uninterpretable dense embeddings fromDPR models into distinct, interpretable latent concepts. |
| Outcome: | The proposed interpretability framework achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches. |
Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation (2022.acl-short)
Copied to clipboard
| Challenge: | Existing sparse retrieval models rely on term-based matching to retrieve relevant documents. |
| Approach: | They propose a framework which augments the representations of documents with interpolation and perturbation. |
| Outcome: | The proposed framework significantly outperforms baselines on the dense retrieval of both the labeled and unlabeled documents. |
Investigating Multi-layer Representations for Dense Passage Retrieval (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Dense retrieval models adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model contain different kinds of linguistic knowledge and behave differently during fine-tuning. |
| Approach: | They propose to utilize representations from multiple encoder layers to make up the representation of a document, which they denote Multi-layer Representations (MLR). |
| Outcome: | The proposed model outperforms dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting and with other advanced training techniques. |
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multilingual and cross-lingual retrieval are lacking in low-resource, morphologically rich languages such as Amharic. |
| Approach: | They propose to train Amharic-specific dense retrieval models based on pre-trained Amharican BERT and RoBERTa backbones. |
| Outcome: | The proposed model achieves 17.6% improvement in MRR@10 and 9.86% gain in Recall@10 over the strongest multilingual baseline, Arctic Embed 2.0. |
SDR: Efficient Neural Re-ranking using Succinct Document Representation (2022.acl-long)
Copied to clipboard
| Challenge: | BERT based ranking models have been successful on various information retrieval tasks, but they are prone to storage and network fetching latency. |
| Approach: | They propose a late-interaction architecture that allows pre-computation of intermediate document representations, thus reducing latency. |
| Outcome: | The proposed model achieves 4x–11.6x higher compression rates on the MSMARCO passage re-reranking task compared to existing methods. |
An Effective Post-training Embedding Binarization Approach for Fast Online Top-K Passage Matching (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing models that learn semantic representations of passages are prone to performance degradation . embedding binarization is a promising branch of model compression . |
| Approach: | They propose an embedding binarization approach that can be used to optimize for online inference. |
| Outcome: | The proposed model can perform query-passage matching acceleration. |
Typo-Robust Representation Learning for Dense Retrieval (2023.acl-short)
Copied to clipboard
Panuthep Tasawong, Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Dense retrieval is a fundamental building block of information retrieval applications. |
| Approach: | They propose a method that aligns misspelled queries with their pristine counterparts to improve contrast between each query and its surrounding queries. |
| Outcome: | The proposed method outperforms the competitors in all cases with misspelled queries. |