F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models and evaluation settings have shortcomings regarding the coupling of answer and explanation which might cause serious issues in user experience. |
| Approach: | They propose a hierarchical model and a new regularization term to strengthen the coupling of answer and explanation and two evaluation scores to quantify the couple. |
| Outcome: | The proposed model strengthens the answer-explanation coupling and provides evaluation scores that align with user experience. |
Similar Papers
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (P19-1)
Copied to clipboard
| Challenge: | Existing question-answering systems are limited in their ability to test reasoning and comprehension. |
| Approach: | They propose a method to automatically extract implications from QA datasets to evaluate models' consistency . they use a heuristic to generate such questions and retrain models with implication-augmented data . |
| Outcome: | The proposed method shows that generated implications are well formed and valid . retraining with implication-augmented data improves consistency on both synthetic and human-generated implications. |
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? (2020.acl-main)
Copied to clipboard
| Challenge: | a new study examines the impact of algorithmic explanations on simulatability of machine learning models . a model is simulatable when a person can predict its behavior on new inputs . |
| Approach: | They conduct human subject tests to isolate effect of algorithmic explanations on simulatability . they find ratings of explanations are not predictive of how helpful they are . |
| Outcome: | The results provide the first reliable estimates of how explanations influence simulatability . they show that ratings are not predictive of how helpful explanations are . |
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
QED: A Framework and Dataset for Explanations in Question Answering (2021.tacl-1)
Copied to clipboard
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, Michael Collins
| Challenge: | Existing question answering systems provide no explanation of reasoning that leads to answer . linguistically informed, extensible framework provides explanations in question answering . |
| Approach: | They propose a linguistically informed, extensible framework for explanations in question answering . they propose an expert-annotated dataset of QED explanations built upon a subset of the Natural Questions dataset . |
| Outcome: | The proposed framework improves the ability of untrained raters to spot errors in QA datasets. |
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)
Copied to clipboard
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
| Challenge: | Existing methods to explain predictions by highlighting salient features are often unstated. |
| Approach: | They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. |
| Outcome: | The proposed framework allows principled, automatic, model-agnostic evaluation of attributions. |
On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratings (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large corpus of domain-expert relevance ratings augments a corpus for compositional explanations . a writer's study shows that the evaluations of compositional inference models underestimate performance . |
| Approach: | They construct a corpus of 126k domain-expert relevance ratings that augment explanations to standardized science exam questions. |
| Outcome: | The results show that evaluations underestimate performance of compositional explanations . they show that models regularly discover and produce valid explanations that are different than gold explanations. |
Model Analysis & Evaluation for Ambiguous Question Answering (2023.findings-acl)
Copied to clipboard
| Challenge: | Ambiguous questions are a challenge for Question Answering models as they require answers that cover multiple interpretations of the original query. |
| Approach: | They aim to investigate whether model/data scaling improves the answers’ quality and whether automated metrics align with human judgment. |
| Outcome: | The proposed models can generate long-form answers that combine conflicting information and provide valuable insights into the limitations of the current approaches. |
On Sample Based Explanation Methods for NLP: Faithfulness, Efficiency and Semantic Evaluation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for explaining "black-box" models such as Influence Functions are becoming more popular. |
| Approach: | They propose a semantic-based evaluation metric that can better align with humans’ judgment of explanations than the widely adopted diagnostic or re-training measures. |
| Outcome: | The proposed method can better align with humans’ judgment of explanations than diagnostic or re-training measures. |
On Evaluating Explanation Utility for Human-AI Decision Making in NLP (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of evidence that explanations help people in situations they are introduced for is a problem in NLP . prior work on explainability has focused on overcoming technical challenges and used proxy evaluations. |
| Approach: | They propose to use existing metrics to evaluate the effectiveness of explanations in NLP . they argue that providing AI predictions does not cause decision makers to speed up work . |
| Outcome: | The proposed evaluations show that providing AI predictions does not cause decision makers to speed up their work without compromising performance. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |