Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to enhance dense retrieval models are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. |
| Approach: | They propose an unsupervised method to enhance passage retrieval in zero-shot settings by iterating a loop that a dense retriever learns from supervision signals provided by a reranker. |
| Outcome: | The proposed method outperforms leading supervised and unsupervised retrievers on the BEIR benchmark while showing strong adaptation abilities to tasks and domains that were unseen during training. |
Similar Papers
Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using mixture-of-memory augmenting to augment language models improves model generalization but with diminishing return. |
| Approach: | They develop a mechanism that augments language models with mixture-of-memory Augmentation (MoMA) they augment strong T5-based retrievers with the option to "plug in" unseen memory at inference time. |
| Outcome: | The proposed model outperforms methods with larger model sizes on the BEIR benchmark and achieves comparable or even better performance than methods relying on target-specific pretraining. |
How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
| Challenge: | Existing techniques to improve dense retrieval suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, some argue due to the limited model capacity. |
| Approach: | They propose to use diverse queries and sources of supervision to train a generalizable DR to achieve high accuracy in both supervised and zero-shot retrieval. |
| Outcome: | The proposed DR can achieve state-of-the-art in supervised and zero-shot evaluations without increasing model size. |
ADAM: Dense Retrieval Distillation with Adaptive Dark Examples (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to retrieve data from multiple encoders are too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student. |
| Approach: | They propose a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with adaptive dark examples. |
| Outcome: | The proposed framework can better transfer the dark knowledge held in the teacher with adaptive dark examples. |
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (2023.emnlp-main)
Copied to clipboard
Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Sultan, Christopher Potts
| Challenge: | Existing methods for information retrieval tasks require large labeled datasets for fine-tuning, but they can experience significant drops in accuracy due to distribution shifts from the training to the target domain. |
| Approach: | They propose a method for using large language models to generate large numbers of synthetic queries cheaply using an expensive LLM. |
| Outcome: | The proposed method boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods. |
Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Dense retrieval (DR) methods first encode texts into a dense embedding space and then conduct text retrieval using efficient nearest neighbor search. |
| Approach: | They propose Momentum adversarial Domain Invariant Representation learning to train a domain classifier that distinguishes source versus target domains and adversarially updates the DR encoder to learn domain invariant representations. |
| Outcome: | The proposed method outperforms baselines on 10+ ranking datasets collected in the BEIR benchmark in the zero-shot setting, with more than 10% relative gains on datasets with enough sensitivity for DR models’ evaluation. |
Boosted Dense Retriever (2022.naacl-main)
Copied to clipboard
| Challenge: | DrBoost is a dense retrieval ensemble that is trained in stages to correct retrieval mistakes . it produces representations which are 4x more compact, while delivering comparable retrieval results. |
| Approach: | They propose a dense retrieval ensemble inspired by boosting that is trained in stages . they produce representations which are 4x more compact, while delivering comparable retrieval results . |
| Outcome: | The proposed model performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. |
Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that adding a instruction tuning stage to training large language models can improve zero-shot task generalization. |
| Approach: | They propose a method that retrieves promptspecific source prompt embeddings from training instances . they train soft prompt embeds for each prompt through prompt tuning and store the samples . |
| Outcome: | The proposed method outperforms hard prompts on unseen tasks by 2.39% points and outperformed 10 out of 11 datasets. |
Towards Robust Neural Retrieval with Source Domain Synthetic Pre-Finetuning (2022.coling-1)
Copied to clipboard
Revanth Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, Avirup Sil
| Challenge: | Existing neural IR systems rely on lexical matching for query-passage alignment, while masked language models use a dual encoder architecture to encode passages and questions into continuous vector representations. |
| Approach: | They propose to enhance the out-of-domain generalization of Dense Passage Retrieval (DPR) through synthetic data augmentation only in the source domain. |
| Outcome: | The proposed model outperforms existing models in in-domain and zero-shot evaluations on Wikipedia-based datasets. |
Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation methods for dense retrieval models use unadapted rerank models, leading to imprecise labels. |
| Approach: | They propose to adapt a rerank model to the target domain before using it for label generation. |
| Outcome: | The proposed model achieves better results across three retrieval datasets. |
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that passage retrieval and passage reranking are important for achieving mutual improvement. |
| Approach: | They propose a unified listwise training approach for passage retrieval and passage reranking that incorporates a retrieval procedure and a hybrid data augmentation strategy. |
| Outcome: | The proposed approach improves on both MSMARCO and Natural Questions datasets. |