OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark for RAG is developed for the financial domain . omnidirectional and automatic benchmarks are difficult to build in vertical domains . |
| Approach: | They propose an omnidirectional and automatic RAG benchmark for the financial domain . they categorize RAG scenarios by task classes and 16 financial topics . |
| Outcome: | The proposed benchmark achieves an 87.47% acceptance ratio in human evaluations of generated instances. |
Similar Papers
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing RAG research focuses on textual data, overlooking rich visual content in financial documents. |
| Approach: | They propose a visual RAG benchmark tailored for finance that integrates multimodal data and provides visual citation to ensure traceability. |
| Outcome: | The proposed visual RAG benchmark integrates multimodal data and provides visual citation to ensure traceability. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |
FinMRAGBench: A Realistic and Complex Benchmark for Multi-Modal RAG in Financial Document Analysis (2026.findings-acl)
Copied to clipboard
Shouqing Yang, Qi Zhang, Yuhang Yang, Ruikang Xu, Yuwei Hou, Zhulin Jia, Lirong Gao, Haobo Wang, Jinglei Chen, Jiexiang Wang, Sheng Guo, Bo Zheng, Gang Chen
| Challenge: | Existing benchmarks for realistic financial analysis fail to capture realistic financial situations involving cross-document retrieval, multi-page evidence integration, and diverse analytical tasks. |
| Approach: | They propose a multi-modal financial RAG benchmark that evaluates large language models in realistic financial analysis settings. |
| Outcome: | The proposed framework achieves the strongest overall performance across all models. |
RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
Xuanwang Zhang, Yun-Ze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
| Challenge: | Existing research on Retrieval Augmented Generation (RAG) does not address the problem of hallucinations and real-time updating of knowledge. |
| Approach: | They propose a modular open-source library to equip LLMs with external knowledge. |
| Outcome: | The proposed approach reduces the need for expensive open-source tools and lacks fair comparisons between novel RAG algorithms. |
DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance. |
| Approach: | They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores. |
| Outcome: | Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs. |
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Evaluating retrieval-augmented generation systems relies on hand annotations for input queries, passages to retrieve, and responses to generate. |
| Approach: | They propose an automated evaluation framework for retrieval-augmented generation (RAG) ARES fine tunes lightweight LLM judges on synthetically generated queries and answers . |
| Outcome: | The proposed framework evaluates RAG systems using only human annotations . it can be used to improve system understanding and create targeted solutions . |
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2024.eacl-demo)
Copied to clipboard
| Challenge: | RAGAs are a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. |
| Approach: | They propose a framework for reference-free evaluation of Retrieval Augmented Generation pipelines. |
| Outcome: | RAGAs can be used to evaluate RAG pipelines without human annotations . the framework can be useful for faster evaluation cycles given the fast adoption of LLMs based on human annotation. |
RAGViz: Diagnose and Visualize Retrieval-Augmented Generation (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) lack domain-specific knowledge and can cause hallucinations. |
| Approach: | They propose a RAG diagnosis tool that visualizes the attentiveness of the generated tokens in retrieved documents. |
| Outcome: | RAGViz provides token and document-level attention visualization and generation comparison upon context document addition and removal. |
RAG-Studio: Towards In-Domain Adaptation of Retrieval Augmented Generation Through Self-Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing RAG systems that use pre-trained LLMs and retrievers often fail in specialized domains and applications. |
| Approach: | They propose a self-aligned training framework that adapts general RAG models to specific domains solely through synthetic data. |
| Outcome: | Experiments on specialized domain corpus, general LLM, and general retriever show that the self-aligned training framework outperforms human-annotated training data in specialized fields. |