What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept. |
| Approach: | They propose a method for automatically building probe datasets from expert knowledge sources, allowing systematic control and a comprehensive evaluation. |
| Outcome: | The proposed model is predisposed to recognize certain types of structural linguistic knowledge, but performance degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy. |
Similar Papers
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (D19-58)
Copied to clipboard
| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical knowledge is encoded in large pre-trained language models (LMs) this is done through supervised classification tasks to predict the grammamatical properties of a span using only the token representations coming from the LM encoder. |
| Approach: | They propose to use a supervised 'edge probing' task to detect grammatical knowledge in large pre-trained language models (LMs) this is done by encoding grammamatical properties using only token representations coming from the LM encoder. |
| Outcome: | The proposed model performs well when fine-tuned or in adversarial situations where the model is forced to learn wrong correlations. |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
Open-Domain Question Answering (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA) |
| Approach: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA . |
| Outcome: | The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods . |
Improving Question Answering with External Knowledge (D19-58)
Copied to clipboard
| Challenge: | ARC-Easy, ARC Challenge, and OpenBookQA use Wikipedia to augment training data . performance degrades when additional instances exhibit higher difficulty than original training data. |
| Approach: | They propose two methods for exploiting external knowledge for QA in science . they enrich the original corpus with relevant text snippets from an open-domain resource . the second method simply increases the amount of training data by appending additional in-domain instances. |
| Outcome: | The proposed methods achieve gains in accuracy of 8.1%, 13.0%, and 12.8% on science QA tasks. |
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work is limited in using small benchmarks with high test-train overlaps. |
| Approach: | They construct a dataset of closed-book QA using SQuAD and investigate the performance of BART. |
| Outcome: | Experiments show that pre-trained language models can achieve high performance on closed-book QA tasks. |
What do we expect from Multiple-choice QA Systems? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that good performance on a dataset might not correlate well with human’s expectations from models that “understand” language. |
| Approach: | They propose to train a top performing multiple choice question answering model against expectations from models that "understand" language. |
| Outcome: | The proposed training paradigm leads to a model that performs on par with the original model while better satisfying our expectations. |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
Do Question Answering Modeling Improvements Hold Across Benchmarks? (2023.acl-long)
Copied to clipboard
| Challenge: | a new study finds that human-constructed and downsampled benchmarks hold more concurrence than downsampled benchmarks. |
| Approach: | They propose to measure concurrence between two QA benchmarks on a set of 20 models . they find that human-constructed benchmarks have high concurrence amongst themselves . |
| Outcome: | The proposed models hold broadly across the diverse landscape of question answering (QA) benchmarks. |