Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) enable zero-shot approaches in open domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. |
| Approach: | They propose to use a distraction-aware answer selection framework to mitigate the impact of irrelevant documents in the retrieved set and the overconfidence of the generated answers to enhance the performance of zero-shot readers. |
| Outcome: | The proposed approach handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. |
Similar Papers
Relevance-assisted Generation for Robust Zero-shot Retrieval (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Despite strong in-domain performance, dense retrievers have shown poor generalization to out-of-domain zero-shot tasks where no training queries are available. |
| Approach: | They propose to generate domain-specific pseudo queries for fine-tuning with domain-relevant relevance between PQ and documents. |
| Outcome: | The proposed approach is more robust to domain shifts, validated on BEIR zero-shot tasks. |
Zero-Shot Rationalization by Multi-Task Transfer Learning from Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract rationales from input text are difficult and impractical. |
| Approach: | They propose a method that leverages multi-task learning and transfer learning to generate rationales through question answering in a zero-shot fashion. |
| Outcome: | The proposed method achieves comparable or even better performance without supervised signal for two benchmark rationalization datasets. |
A Deep Relevance Model for Zero-Shot Document Filtering (P18-1)
Copied to clipboard
| Challenge: | Existing methods for document classification do not consider document filtering . existing methods do not include document filter. |
| Approach: | They propose a deep relevance model for zero-shot document filtering called DAZER . they use word embeddings to extract the relevance signals from word embeds . |
| Outcome: | The proposed model outperforms existing models on two document collections . it estimates the relevance between a document and a category by using seed words . |
Precise Zero-Shot Dense Retrieval without Relevance Labels (2023.acl-long)
Copied to clipboard
| Challenge: | Existing dense retrieval systems that use semantic embedding similarities can be effective across tasks and languages. |
| Approach: | They propose to pivot through Hypothetical Document Embeddings (HyDE) given a query, HyDE first zero-shot prompts an instruction-following language model to generate a hypothetical document. |
| Outcome: | The proposed method significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers across tasks and languages. |
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs cannot generalize to domain-specific parsing tasks in a zero-shot setting. |
| Approach: | They propose a task-oriented parsing method that decomposes parse problem into abstractive and extractive question-answering problems. |
| Outcome: | The proposed method decomposes a parsing problem into abstractive and extractive question-answering (QA) problems. |
Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing pointwise LLMs provide noisy or biased answers for documents that are partially relevant to the query. |
| Approach: | They propose to incorporate fine-grained relevance labels into the LLM prompt . they propose to better differentiate between documents with different levels of relevance . |
| Outcome: | The proposed model can differentiate between documents with different levels of relevance to the query and derive a more accurate ranking. |
Self-Prompting Large Language Models for Zero-Shot Open-Domain QA (2024.naacl-long)
Copied to clipboard
| Challenge: | Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing specific background documents. |
| Approach: | They propose a framework to explicitly utilize the massive knowledge encoded in LLM parameters and their strong instruction understanding abilities. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on three widely-used ODQA datasets and achieves comparable performance with customized fine-tuned models on full training data. |
Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that large language models (LLMs) rank documents based on the probability of generating the query given the content of a document. |
| Approach: | They propose a ranking system that integrates LLMs with a hybrid zero-shot retriever. |
| Outcome: | The proposed system shows exceptional ranking in both zero-shot and few-shot scenarios. |
A Thorough Examination on Zero-shot Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
Ruiyang Ren, Yingqi Qu, Jing Liu, Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, Ji-Rong Wen
| Challenge: | Recent advances in dense retrieval (DR) models have been shown to be not as competitive as traditional sparse retrieval models in a zero-shot retrieval setting. |
| Approach: | They propose to examine the zero-shot capability of DR models by analyzing key factors related to source training set and potential bias from target dataset. |
| Outcome: | The proposed model is not as competitive as sparse retrieval models in a zero-shot retrieval setting. |
Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot summarization of abstractive summaries for given articles, but little is known about their robustness at this task. |
| Approach: | They propose a strategy that uses the most relevant sentences to generate an ideal summary and then paraphrases them to obtain a minimally perturbed dataset. |
| Outcome: | The proposed approach can be used to measure the robustness of LLMs as summarizers on a minimally perturbed dataset. |