Papers with multi-hop
Towards Reliable Agents: Benchmarking Customized LLM-Based Retrieval-Augmented Generation Frameworks with Deployment Validation (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks for general-purpose RAG systems, such as CRAG, RGB, MultiHop-RAG, and CRUD-RAGG, are limited and lack a benchmark specifically tailored to evaluate frameworks. |
| Approach: | They evaluated OpenAI’s Assistants API versus a RAG assistant built with Langchain and deployed a system based on benchmark insights as a course assistant over a two-year span. |
| Outcome: | The proposed benchmarks show that domain-specific retrieval impacts response accuracy and highlight key challenges in real-world deployment. |
ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing benchmarks assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. |
| Approach: | They propose a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. |
| Outcome: | The proposed framework evaluates end-to-end tool orchestration by agents in hierarchical Model-Context Protocol (MCP) environments. |
A Divide-And-Conquer Approach for Multi-label Multi-hop Relation Detection in Knowledge Base Question Answering (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for relation detection only detect one path to obtain the answer without considering other correct paths. |
| Approach: | They propose a divide-and-conquer approach for multi-label multi-hop relation detection . they propose 'path sampling mechanism' to generate diverse relation paths . |
| Outcome: | The proposed approach outperforms other competitive approaches on the FreebaseQA benchmark dataset. |
BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for multi-hop reasoning in biomedical domain are lacking . bioHopR provides benchmarks to evaluate multi-step reasoning in structured biomedic knowledge graphs . |
| Approach: | They propose a benchmark to evaluate multi-hop, multi-answer reasoning in biomedical knowledge graphs. |
| Outcome: | BioHopR evaluates multi-hop reasoning in biomedical knowledge graphs based on the PrimeKG model . it outperforms proprietary models and open-source biomedal models in 1-hop and 2-hop tasks . |
Measuring Retrieval Complexity in Question Answering Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a new metric, retrieval complexity (RC), measures the difficulty of answering questions. |
| Approach: | They propose a retrieval complexity metric conditioned on the completeness of retrieved documents . they propose an unsupervised pipeline to measure RC given an arbitrary retrieval system . |
| Outcome: | The proposed pipeline measures RC more accurately than alternative estimators on six challenging QA benchmarks. |
SCRIBE: Structured Chain Reasoning for Interactive Behaviour Explanations using Tool Calling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models can be used to provide personalized feedback in educational settings, but they face privacy concerns, limited computational resources, and the need for pedagogically valid responses. |
| Approach: | They propose a framework for multi-hop, tool-augmented reasoning to generate valid responses to student questions about feedback reports. |
| Outcome: | The proposed framework can generate valid responses to student questions about feedback reports using domain-specific tools and self-reflective inference pipelines. |