Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods often fail to produce objective results and favor high similarity to the ground-truth question-answer pairs. |
| Approach: | They propose an alternative approach to evaluate question-answer generation using Generative Interpretation (GI) GI outperforms existing evaluation methods in terms of human alignment . |
| Outcome: | The proposed approach outperforms existing evaluation methods in human alignment and shows comparable performance with GPT3.5, only with BART-large. |
Similar Papers
Asking Questions Like Educational Experts: Automatically Generating Question-Answer Pairs on Real-World Examination Data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate high quality question-answer pairs are limited . a new framework is proposed for the question-answer generation task on real-world examination data. |
| Approach: | They propose a multi-agent communication model to generate and optimize the question and keyphrases iteratively and then apply the generated question and keys to guide the generation of answers. |
| Outcome: | The proposed framework makes great breakthroughs in the question-answer pair generation task. |
Elaboration-Generating Commonsense Question Answering at Scale (2023.acl-long)
Copied to clipboard
| Challenge: | elaborations are generated using language models that generate background knowledge that helps improve performance . human evaluations show that the quality of the generated ellaborations is high . |
| Approach: | They propose to finetune smaller language models to generate useful intermediate context . they compare a language model with an answer predictor and generate elaborations . human evaluations show that the quality of the generated ellaborations is high . |
| Outcome: | The proposed framework outperforms other models on commonsense questions on four commons sense benchmarks. |
Generating Responses that Reflect Meta Information in User-Generated Question Answer Pairs (2020.lrec-1)
Copied to clipboard
Takashi Kodama, Ryuichiro Higashinaka, Koh Mitsuda, Ryo Masumura, Yushi Aono, Ryuta Nakamura, Noritake Adachi, Hidetoshi Kawabata
| Challenge: | Existing approaches to realize consistent personalities require expensive data collection. |
| Approach: | They propose to collect question-answer pairs for particular characters from online users . meta information such as emotion and intimacy was also collected . |
| Outcome: | The proposed method can be used to train neural conversational models with high quality questions and meta information. |
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
Evaluation of Question Answer Generation for Portuguese: Insights and Datasets (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Automatic question generation is an increasingly important task that can be applied in educational settings, data augmentation for question-answering (QA), and conversational systems. |
| Approach: | They adapt and apply QAG approaches to generate question-answer pairs given context and look into strategies for error filtering and their effects. |
| Outcome: | The proposed methods can generate question-answer pairs in Portuguese, a widely spoken language that is underrepresented in natural language processing research. |
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books (2022.acl-long)
Copied to clipboard
| Challenge: | Existing question answering (QA) techniques are created mainly to answer questions asked by humans, but in educational applications, teachers often need to decide what questions to ask . |
| Approach: | They propose to use a fairytale-themed storybook as input to generate QA pairs that can test a student's comprehension skills. |
| Outcome: | The proposed system outperforms state-of-the-art QAG baseline systems and builds an interactive story-telling application for the future real-world deployment. |
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021.acl-long)
Copied to clipboard
| Challenge: | evaluators distinguish between human- and machine-authored text in three domains without training . evals' accuracy improved up to 55%, but it did not significantly improve across the three domain. |
| Approach: | They examine the role untrained human evaluations play in NLG evaluation and propose ways to improve their evaluations. |
| Outcome: | The evaluators distinguished between human- and machine-authored text at random chance level without training, but their accuracy did not improve across the three domains. |
Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment (2023.acl-short)
Copied to clipboard
Roni Rabin, Alexandre Djerbetian, Roee Engelberg, Lidan Hackmon, Gal Elidan, Reut Tsarfaty, Amir Globerson
| Challenge: | Human communication often involves information gaps between the interlocutors. |
| Approach: | They propose a model that generates such gap-focused questions automatically . they propose an evaluation by human annotators of the generated questions . |
| Outcome: | The proposed model outperforms human generated questions in a competitive environment. |
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (P19-1)
Copied to clipboard
| Challenge: | Existing question-answering systems are limited in their ability to test reasoning and comprehension. |
| Approach: | They propose a method to automatically extract implications from QA datasets to evaluate models' consistency . they use a heuristic to generate such questions and retrain models with implication-augmented data . |
| Outcome: | The proposed method shows that generated implications are well formed and valid . retraining with implication-augmented data improves consistency on both synthetic and human-generated implications. |
A Feasibility Study of Answer-Agnostic Question Generation for Education (2022.findings-acl)
Copied to clipboard
Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, DaHyeon Choi, Chuning Yuan, Chris Callison-Burch
| Challenge: | a feasibility study into the applicability of answer-agnostic question generation models to textbook passages is conducted . a significant portion of errors arise from asking irrelevant or un-interpretable questions, a study finds . |
| Approach: | They conduct a feasibility study into the applicability of answer-agnostic question generation models to textbook passages. |
| Outcome: | The proposed model reduces the time it takes to write questions that target salient concepts . the proposed model would help professors write quizzes faster and help students stay engaged . |