SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are prone to hallucination and rely on static, pre-annotated references for evaluation. |
| Approach: | They propose a framework to assess large language models without fixed ground-truth answers by iteratively generating web queries and synthesizing external evidence. |
| Outcome: | The proposed framework achieves substantial to perfect agreement with human evaluations on multiple free-form QA benchmarks. |
Similar Papers
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024.findings-acl)
Copied to clipboard
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong
| Challenge: | Modern large language models often "hallucinate" plausible but factually incorrect information, which reduces their trustworthiness especially in settings where accurate and up-to-date information is critical. |
| Approach: | They develop a human evaluation procedure to measure correctness and hallucination and use it to benchmark both closed and open-source LLMs. |
| Outcome: | The proposed method outperforms both competing search engine-augmented prompting methods and commercial systems on search-augmented QA. |
Over-Searching in Search-Augmented Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. |
| Approach: | They conduct a systematic evaluation of over-searching across multiple dimensions including query types, model categories, retrieval conditions, and multi-turn conversations. |
| Outcome: | The proposed model improves answer accuracy on answerable queries but harms abstention on unanswerable ones . |
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) generate factually incorrect answers based on their parametric memory. |
| Approach: | They propose a retrieval-augmented large language model that can dynamically select the most suitable strategy based on query complexity. |
| Outcome: | The proposed approach improves the performance of QA systems on open-domain QA datasets. |
How Credible Is an Answer From Retrieval-Augmented LLMs? Investigation and Evaluation With Multi-Hop QA (2025.coling-main)
Copied to clipboard
| Challenge: | Retrieval-augmented large language models (RaLLMs) are reshaping knowledge acquisition, offering long-form, knowledge-grounded answers through advanced reasoning and generation capabilities. |
| Approach: | They propose a benchmarking system to evaluate RaLLMs' correctness and Groundedness to determine their reliability in multi-hop question-answering tasks. |
| Outcome: | The proposed model-based evaluation pipeline for multi-hop question-answering tasks reveals that the model generates inaccuracies when dealing with flawed or partial knowledge. |
Automatic Evaluation of Attribution by Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) incorporate external references to generate and support claims. however, evaluating the attribution remains an open problem. |
| Approach: | They investigate automatic evaluation of attribution given by large language models . they define different types of attributed errors and then explore two approaches . |
| Outcome: | The proposed methods highlight promising signals and challenges. |
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
Estimating Knowledge in Large Language Models Without Generating a Single Token (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate knowledge in large language models require querying and evaluating the model's generated responses. |
| Approach: | They ask whether it is possible to estimate how knowledgeable a model is about a subject entity only from its internal computation. |
| Outcome: | The proposed model performs well with QA accuracy and FActScore . it can be leveraged to guide decisions on how to apply further training or augment queries with retrieval. |
FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models (2024.emnlp-demo)
Copied to clipboard
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, Shikun Zhang
| Challenge: | Large language models (LLMs) have revolutionized natural language processing with impressive performance across various tasks. |
| Approach: | They propose a framework for automated evaluations of large language models . they open-source their code at https://github.com/WisdomShell/FreeEval . |
| Outcome: | The framework is open-source and can be used to develop and validate new evaluation methods. |
RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models (2023.emnlp-demo)
Copied to clipboard
Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka, Osamu Torii, Jun Deguchi
| Challenge: | Existing libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes. |
| Approach: | They propose an open-source framework to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. |
| Outcome: | The framework improves hand-crafted prompts, inference processes and quantitatively measures overall system performance. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |