Papers by Keshav Santhanam
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023.findings-acl)
Copied to clipboard
Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts
| Challenge: | Neural information retrieval (IR) systems have progressed rapidly in recent years . many IR benchmarks focus on downstream task accuracy, concealing costs incurred . |
| Approach: | They propose to include efficiency considerations on IR benchmarks to help drive progress . eral et al. propose to incorporate query latency and cost budgets into evaluation . |
| Outcome: | a new study shows that the best IR system varies according to how efficiency considerations are chosen and weighed . the proposed benchmarks would allow for more thorough exploration of possible system designs . |
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (2023.emnlp-main)
Copied to clipboard
Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Sultan, Christopher Potts
| Challenge: | Existing methods for information retrieval tasks require large labeled datasets for fine-tuning, but they can experience significant drops in accuracy due to distribution shifts from the training to the target domain. |
| Approach: | They propose a method for using large language models to generate large numbers of synthetic queries cheaply using an expensive LLM. |
| Outcome: | The proposed method boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods. |
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural information retrieval (IR) methods encode queries and documents into single vectors, but late interaction models produce multi-vector representations at the granularity of each token. |
| Approach: | They propose a retrieval method that couples an aggressive residual compression mechanism with a denoised supervision strategy to improve the quality and space footprint of late interaction. |
| Outcome: | The proposed retriever improves quality and space footprint of late interaction models while reducing space footprint by 6–10x. |