QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics fail to align well with human judgments when evaluating QG questions. |
| Approach: | They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions. |
| Outcome: | The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency . |
Similar Papers
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)
Copied to clipboard
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics. |
| Approach: | They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer. |
| Outcome: | The proposed evaluator improves on three typical NLG tasks and improves with external knowledge. |
Reference-based Metrics Disprove Themselves in Question Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for question generation are based on human-written references . however, the results of the metrics on our study disprove the metrics themselves . |
| Approach: | They propose to use human-written references to evaluate question generation . they propose to combine criteria such as naturalness, answerability, and complexity . |
| Outcome: | The proposed model is based on multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models. |
Towards a Better Metric for Evaluating Question Generation Systems (D18-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics based on n-gram similarity do not correlate well with human judgments . large datasets for document Question Answering (QA) have enabled the development of end-to-end supervised models . |
| Approach: | They propose a scoring function to capture answerability of questions . they also integrate existing similarity metrics into the scoring function . |
| Outcome: | The proposed scoring function improves human judgments on question answerability . the proposed scoring functions are made publicly available . |
Evaluation of Question Generation Needs More References (2023.findings-acl)
Copied to clipboard
Shinhyeok Oh, Hyojun Go, Hyeongdon Moon, Yunsung Lee, Myeongho Jeong, Hyun Seung Lee, Seungtaek Choi
| Challenge: | Existing evaluations of QG methods rely on single reference-based similarity metrics . multiple (pseudo) references are more effective for QG evaluation . |
| Approach: | They propose to paraphrase the reference question for a more robust QG evaluation. |
| Outcome: | The proposed frameworks show higher correlation with human evaluations than evaluation with a single reference. |
QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for assessing question generation fail to take into account the input context of generation. |
| Approach: | They propose a context-aware Relevance evaluation metric for Question Generation that takes into account the context of question generation into account. |
| Outcome: | The proposed metric achieves higher correlation with human judgments while being much more robust to adversarial samples. |
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on automatic story generation (ASG) rely on human criteria, but there is little research on how well they correlate with human criteria. |
| Approach: | They propose to use human criteria to evaluate automatic story generation (ASG) their paper proposes to use HANNA to quantitatively evaluate correlations between 72 automatic metrics and human criteria. |
| Outcome: | The proposed model compared human criteria with automatic criteria and found that they were significantly better than human criteria. |
Exploring Question-Specific Rewards for Generating Deep Questions (2020.coling-main)
Copied to clipboard
| Challenge: | Recent question generation approaches use the sequence-to-sequence framework to optimize the log likelihood of ground-truth questions using teacher forcing. |
| Approach: | They propose to optimize for QG-specific objectives via reinforcement learning to improve question quality. |
| Outcome: | The proposed model improves the fluency, relevance, and answerability of generated questions. |
Generative Language Models for Paragraph-Level Question Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Powerful generative models have led to recent progress in question generation. |
| Approach: | They propose a multilingual and multidomain benchmark for question generation that unifies existing datasets by converting them to a standard QG setting. |
| Outcome: | The proposed benchmark unifies existing question answering datasets to a QG setting. |
LaMP-QA: A Benchmark for Personalized Long-form Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalization in question answering systems remains underexplored due to lack of resources . a new benchmark for personalized answer generation is being developed . |
| Approach: | They propose a LaMP-QA benchmark for evaluating personalized answer generation . they compare evaluation strategies for generating personalized responses and measure their alignment with human preferences. |
| Outcome: | The proposed benchmark shows that incorporating the personalized context provided leads to performance improvements. |
Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Question generation models are often evaluated with standardized NLG metrics that are based on n-gram overlap. |
| Approach: | They propose to use QGen to help teachers automate the generation of reading comprehension quizzes by comparing n-gram overlap with BLEU to compare system-generated questions with heldout human-written references. |
| Outcome: | The best model had only 68.4% of its questions accepted by the ten teachers who participated in the study. |