Papers with GRADE
GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current evaluations of RAG systems overlook structural complexity and multi-step reasoning . GRADE model enables fine-grained analysis of Ragging performance . |
| Approach: | They propose a framework that models retrieval difficulty along two orthogonal dimensions . they extract knowledge graphs and augment them through semantic clustering to recover missing links . |
| Outcome: | The proposed framework models retrieval difficulty along two orthogonal dimensions . error rates correlate with the framework, and it validates its diagnostic utility. |
GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics only consider surface features or utterance-level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. |
| Approach: | They propose a graph-enhanced evaluation metric GRADE to evaluate dialogue coherence . GRADE incorporates utterance-level contextualized representations and fine-grained topic-level graph representations to improve communication logic. |
| Outcome: | The proposed evaluation metric outperforms state-of-the-art metrics on measuring diverse dialogue models in terms of Pearson and Spearman correlations with human judgments. |
IM^2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics for dialogue systems are expensive and time-consuming . current evaluation metrics focus on a single quality or several qualities . |
| Approach: | They propose an interpretable, multi-faceted, and controllable framework to combine dialogue metrics which are good at measuring different qualities. |
| Outcome: | The proposed framework integrates a large number of evaluation metrics to improve the performance of the model. |