Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Question generation models are often evaluated with standardized NLG metrics that are based on n-gram overlap. |
| Approach: | They propose to use QGen to help teachers automate the generation of reading comprehension quizzes by comparing n-gram overlap with BLEU to compare system-generated questions with heldout human-written references. |
| Outcome: | The best model had only 68.4% of its questions accepted by the ten teachers who participated in the study. |
Similar Papers
Towards a Better Metric for Evaluating Question Generation Systems (D18-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics based on n-gram similarity do not correlate well with human judgments . large datasets for document Question Answering (QA) have enabled the development of end-to-end supervised models . |
| Approach: | They propose a scoring function to capture answerability of questions . they also integrate existing similarity metrics into the scoring function . |
| Outcome: | The proposed scoring function improves human judgments on question answerability . the proposed scoring functions are made publicly available . |
SkillQG: Learning to Generate Question for Reading Comprehension Assessment (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing question generation systems focus on the literal nature of questions and rarely consider comprehension types of the generated questions. |
| Approach: | They propose a question generation framework with controllable comprehension types for machine reading comprehension models. |
| Outcome: | Empirical results show that SkillQG outperforms baselines in quality, relevance, and skill-controllability while showing a performance boost in downstream question answering task. |
Vocabulary Matters: A Simple yet Effective Approach to Paragraph-level Question Generation (2020.aacl-main)
Copied to clipboard
| Challenge: | Current neural network-based questions generation techniques take only one or two sentences as input. |
| Approach: | They propose a simple yet effective technique for question generation from paragraphs . they augment a sequence-to-sequence QG model with dynamic, paragraph-specific dictionary . |
| Outcome: | The proposed model outperforms state-of-the-art systems in question generation from paragraphs in automatic and human evaluation. |
Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs (2022.naacl-main)
Copied to clipboard
| Challenge: | NLP-powered automatic question generation (QG) techniques have not been widely adopted in classrooms to date. |
| Approach: | They propose to identify key impediments and improve the usability of NLP-powered automatic question generation techniques by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models. |
| Outcome: | The proposed methods can be used by 11 instructors across 7 universities and highlight their needs and needs when creating questions. |
ConQuer: A Framework for Concept-Based Quiz Generation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing concepts-based quiz generation frameworks that leverage external knowledge sources are challenging and labor intensive. |
| Approach: | They propose a concept-based quiz generation framework that leverages external knowledge sources to assess the quality of the generated quizzes, using LLMs as judges. |
| Outcome: | The proposed framework shows a 4.8% improvement in evaluation scores and a 77.52% win rate in pairwise comparisons against baseline quiz sets. |
Exploring Question-Specific Rewards for Generating Deep Questions (2020.coling-main)
Copied to clipboard
| Challenge: | Recent question generation approaches use the sequence-to-sequence framework to optimize the log likelihood of ground-truth questions using teacher forcing. |
| Approach: | They propose to optimize for QG-specific objectives via reinforcement learning to improve question quality. |
| Outcome: | The proposed model improves the fluency, relevance, and answerability of generated questions. |
Evaluation of Question Generation Needs More References (2023.findings-acl)
Copied to clipboard
Shinhyeok Oh, Hyojun Go, Hyeongdon Moon, Yunsung Lee, Myeongho Jeong, Hyun Seung Lee, Seungtaek Choi
| Challenge: | Existing evaluations of QG methods rely on single reference-based similarity metrics . multiple (pseudo) references are more effective for QG evaluation . |
| Approach: | They propose to paraphrase the reference question for a more robust QG evaluation. |
| Outcome: | The proposed frameworks show higher correlation with human evaluations than evaluation with a single reference. |
Automatic Distractor Generation for Multiple Choice Questions in Standard Tests (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to generate distractors for multiple choice questions are expensive and time-consuming. |
| Approach: | They propose a question and answer guided distractor generation framework to automate distractors generated by domain experts. |
| Outcome: | The proposed model outperforms existing models and achieves state-of-the-art on a large-scale dataset. |
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics fail to align well with human judgments when evaluating QG questions. |
| Approach: | They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions. |
| Outcome: | The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency . |
A Practical Toolkit for Multilingual Question and Answer Generation (2023.acl-demo)
Copied to clipboard
| Challenge: | Generating questions and answers from text is a challenging task due to the expected structured output. |
| Approach: | They propose an online service for multilingual QAG along with a python package for model fine-tuning, generation, and evaluation. |
| Outcome: | The proposed model is available in eight languages and can be used online or locally via lmqg. |