Multi-Layered Evaluation Using a Fusion of Metrics and LLMs as Judges in Open-Domain Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for comparing machine-generated answers with reference are not perfect in terms of accuracy or cost. |
| Approach: | They propose to summarize long answers and use shortened versions to improve evaluation . they propose a multi-layered evaluation methodology that integrates different metrics tailored to various scenarios . |
| Outcome: | The proposed method outperforms existing evaluation methods but is more cost-effective than existing methods. |
Similar Papers
Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation models fail to identify lexical matching failures for open-domain question answering. |
| Approach: | They manually evaluate open-domain QA models by manually evaluating their answers on a popular benchmark. |
| Outcome: | The proposed model performs better on NQ-open than existing models and more than 50% of lexical matching failures are attributed to semantically equivalent answers. |
AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Question answering (QA) tasks have been extensively studied in the field of natural language processing. |
| Approach: | They propose a method that leverages large language models and the analytic hierarchy process to assess open-ended questions. |
| Outcome: | The proposed method more closely aligns with human judgment compared to baselines on four datasets. |
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are expensive and time-consuming. |
| Approach: | They propose a framework that utilizes LLMs to generate synthetic evaluation datasets . they propose meta-correlation to measure alignment between metric rankings and human benchmarks based on synthetic data . |
| Outcome: | The proposed framework achieves meta-correlations exceeding 0.9 in multilingual QA and replaces human judgment with synthetic evaluation datasets. |
VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for image captioning are primarily designed for short captions and are not suitable for long captions. |
| Approach: | They propose an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. |
| Outcome: | The proposed metric outperforms existing metrics and achieves superhuman performance on LongCap-Arena. |
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions (2026.findings-acl)
Copied to clipboard
| Challenge: | LLM-as-a-Judge uses LLMs to evaluate open-ended questions . however, the discrepancy between LLM generated evaluations and human evaluations remains a critical problem in this field . |
| Approach: | They propose a framework that orchestrates evaluations across multiple criteria using multiple LLMs. |
| Outcome: | The proposed framework achieves superior alignment with human evaluations compared to baselines. |
Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models generate human-like text, making them unreliable for biomedical relation extraction tasks. |
| Approach: | They propose to use Large Language Models as judges to evaluate biomedical relation extraction . they propose structured output formatting for LLM-generated responses that helps LLMs improve their performance by 15%. |
| Outcome: | The proposed method improves LLM-Judges' performance by 15% . it is cheaper and more efficient than human evaluation metrics, the authors say . |
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing meta-evaluation benchmarks are static, outdated, and lacking in multilingual coverage. |
| Approach: | They propose a framework for curating more representative open-domain dialogue evaluation benchmarks . they leverage several LLMs to generate user-chatbot multilingual dialogues conditioned on varied seed contexts based on a state-of-the-art LLM . |
| Outcome: | The proposed framework exploits state-of-the-art LLMs to perform multilingual evaluations of open-domain chatbots. |
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment (2026.acl-short)
Copied to clipboard
| Challenge: | Evaluating Question Answering systems in low-resource Indic languages remains challenging due to the scarcity of annotated data and the lack of reliable evaluation metrics. |
| Approach: | They propose a language-based multi-aspect evaluation framework for question answering systems . the framework integrates semantic similarity, factual completeness, numerical accuracy and contextual relevance . |
| Outcome: | The proposed metric is evaluated across eight Indic-language QA tasks using multiple LLMs . Across all settings, it shows stronger agreement with human evaluation . |
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts (2024.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for the automated evaluation of natural language texts are based on a large language model (LLM) that fails to agree with human judges and is not fully validated by the human judges. |
| Approach: | They propose a large language model (LLM) that generates a distribution over potential responses to assess multiple dimensions of interest. |
| Outcome: | The proposed framework predicts human judges' assessment of user satisfaction on a scale of 1–4 with an RMS error 0.5, a 2 improvement over the uncalibrated baseline. |