Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering (2024.lrec-main)
Copied to clipboard
Yuan Gao, Yiheng Zhu, Yuanbin Cao, Yinzhi Zhou, Zhen Wu, Yujie Chen, Shenglan Wu, Haoyuan Hu, Xinyu Dai
| Challenge: | Open Domain Multi-Hop Question Answering (ODMHQA) is one of the most challenging tasks in Natural Language Processing (NLP) |
| Approach: | They propose a mechanism that leverages the intrinsic capabilities of Large Language Models to judge whether the generated answers are off-topic. |
| Outcome: | The proposed method reduces the occurrence of off-topic answers by nearly 13%, improving the performance in Exact Match (EM) by nearly 3% compared to the baseline method without the Dr3 mechanism. |
Similar Papers
Over-Searching in Search-Augmented Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. |
| Approach: | They conduct a systematic evaluation of over-searching across multiple dimensions including query types, model categories, retrieval conditions, and multi-turn conversations. |
| Outcome: | The proposed model improves answer accuracy on answerable queries but harms abstention on unanswerable ones . |
Exploring Language Model Generalization in Low-Resource Extractive QA (2025.coling-main)
Copied to clipboard
| Challenge: | Existing LLMs struggle with dataset demands of closed domains such as medicine and law . current LLM performance in closed domain is lacking, even on traditional tasks such as Natural Language Inference . |
| Approach: | They investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift . they find that LLMs struggle with dataset demands of closed domains . |
| Outcome: | The proposed model performs poorly in extractive question answering tasks under domain drift . the proposed model can generalize to domains that require specific knowledge without training . |
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open-domain question-answering methods lack quality assurance . existing methods lack scalability and poor diversity, hindering LLMs' capabilities . |
| Approach: | They propose an open-domain multi-hop reasoning framework to answer multi-choice questions . they propose an adaptive sampler for in-context selection and self-prompted inference . |
| Outcome: | The proposed framework surpasses the existing SOTA methods on large-scale datasets and doubles the zero-shot performance of small-scale LLMs. |
Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent approaches to multi-hop question answering rely on in-context learning . however, these models contain billions of parameters making them inefficient at inference time. |
| Approach: | They propose a framework that allows for improving smaller language models with less than 10 human-annotated QA pairs by synthesizing millions of multi-hop questions and claims to fine tune language models. |
| Outcome: | The proposed framework improves model performance on multi-hop question answering and fact verification benchmarks while being almost one-third the size in parameter count. |
Simple yet Effective Bridge Reasoning for Open-Domain Multi-Hop Question Answering (D19-58)
Copied to clipboard
| Challenge: | Existing work on open-domain multi-hop question answering relies on off-the-shelf information retrieval techniques to retrieve answer passages. |
| Approach: | They propose a new subproblem for open-domain multi-hop question answering . they aim to recognize the anchor from a set of start passages with a reading comprehension model . |
| Outcome: | The proposed method significantly improves the baseline method on the open-domain hotpotQA benchmark. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-hop Question Answering (MHQA) adds layers of complexity to question answering tasks. |
| Approach: | They explore how LMs respond to multi-hop questions by permuting search results under various configurations. |
| Outcome: | The proposed model outperforms decoder-only models in MHQA tasks despite being significantly smaller in size . |
Offloaded Reasoning: Efficient Inference for Large Language Models via Modular Reasoning and Refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate strong reasoning capabilities but are expensive to run at inference time, limiting their practical deployment. |
| Approach: | They propose Offloaded Reasoning, a modular strategy where a lightweight model generates intermediate reasoning traces that are then used by a larger model to produce the final answer. |
| Outcome: | The proposed approach achieves faster inferences than full large-model reasoning with minimal accuracy loss while recovering or exceeding full accuracy at substantially lower cost. |
ODUTQA-MDC: A Task for Open-Domain Underspecified Tabular QA with Multi-turn Dialogue-based Clarification (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to tabular QA are limited to closed-domain scenarios . existing approaches do not solve the core challenge of generating correct answers without user clarification . |
| Approach: | They propose a benchmark to tackle underspecified or uncertain queries in tabular question answering . they propose ODUTQA-MDC task and a multi-agent framework to detect ambiguities . |
| Outcome: | The proposed framework excels at detecting ambiguities, clarifying them through dialogue, and refining answers. |
Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step problems, but little is known about how these computations are implemented internally. |
| Approach: | They propose a "back-patching" analysis method to solve multi-hop queries . they propose resolving the bridge entity into the bridge and the second hop into the target entity into latent steps. |
| Outcome: | The proposed method solves multi-hop queries that require two information extraction steps . it shows that the later layers lack the necessary knowledge to correctly generate the answer . |