What’s in a Name? Answer Equivalence For Open-Domain Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | A flaw in QA evaluation is that annotations often only provide one answer . therefore, model predictions semantically equivalent to the answer but superficially different are considered incorrect. |
| Approach: | They explore using alias entities from knowledge bases to extract additional answers . they incorporate additional answers for evaluation and model training with equivalent answers based on the results . |
| Outcome: | The proposed solution improves the accuracy of evaluation with additional answers and improves model training with equivalent answers. |
Similar Papers
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
Return of EM: Entity-driven Answer Set Expansion for QA Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that using large language models (LLMs) is the most reliable method to evaluate QA models, but suffers from limited interpretability, high cost, and environmental harm. |
| Approach: | They propose to use soft exact match (EM) with entity-driven answer set expansion to expand gold answer set to include diverse surface forms. |
| Outcome: | The proposed method outperforms traditional evaluation methods while offering the benefits of high interpretability and reduced environmental harm. |
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
Improving Question Answering with External Knowledge (D19-58)
Copied to clipboard
| Challenge: | ARC-Easy, ARC Challenge, and OpenBookQA use Wikipedia to augment training data . performance degrades when additional instances exhibit higher difficulty than original training data. |
| Approach: | They propose two methods for exploiting external knowledge for QA in science . they enrich the original corpus with relevant text snippets from an open-domain resource . the second method simply increases the amount of training data by appending additional in-domain instances. |
| Outcome: | The proposed methods achieve gains in accuracy of 8.1%, 13.0%, and 12.8% on science QA tasks. |
QUADRo: Dataset and Models for QUestion-Answer Database Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tools do not consider answers (question-question similarity only) or their quality in the annotation process. |
| Approach: | They propose an open-domain annotated resource that retrieves questions from pre-computed databases . they use a binary label associated with each question/answer pair to determine relevance . |
| Outcome: | The proposed model trains and evaluates models on 443,000 examples with respect to key aspects of QA systems. |
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (P19-1)
Copied to clipboard
| Challenge: | Existing question-answering systems are limited in their ability to test reasoning and comprehension. |
| Approach: | They propose a method to automatically extract implications from QA datasets to evaluate models' consistency . they use a heuristic to generate such questions and retrain models with implication-augmented data . |
| Outcome: | The proposed method shows that generated implications are well formed and valid . retraining with implication-augmented data improves consistency on both synthetic and human-generated implications. |
Accurate and Nuanced Open-QA Evaluation Through Textual Entailment (2024.findings-acl)
Copied to clipboard
| Challenge: | Open-domain question answering (Open-QA) evaluations are criticized for the ambiguity in questions and the lack of semantic understanding in evaluators. |
| Approach: | They propose to examine the entailment relations of answers to identify more informative and more general system answers. |
| Outcome: | The proposed evaluations offer a much closer evaluation to human judgment on NaturalQuestions and TriviaQA while being learning-free. |
Open-Domain Question Answering (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA) |
| Approach: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA . |
| Outcome: | The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods . |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation models fail to identify lexical matching failures for open-domain question answering. |
| Approach: | They manually evaluate open-domain QA models by manually evaluating their answers on a popular benchmark. |
| Outcome: | The proposed model performs better on NQ-open than existing models and more than 50% of lexical matching failures are attributed to semantically equivalent answers. |