| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
Similar Papers
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
Do Question Answering Modeling Improvements Hold Across Benchmarks? (2023.acl-long)
Copied to clipboard
| Challenge: | a new study finds that human-constructed and downsampled benchmarks hold more concurrence than downsampled benchmarks. |
| Approach: | They propose to measure concurrence between two QA benchmarks on a set of 20 models . they find that human-constructed benchmarks have high concurrence amongst themselves . |
| Outcome: | The proposed models hold broadly across the diverse landscape of question answering (QA) benchmarks. |
What Question Answering can Learn from Trivia Nerds (2020.acl-main)
Copied to clipboard
| Challenge: | a question answering dataset is a competition that has a leaderboard that determines the best answers. |
| Approach: | They propose to apply the best practices of trivia tournaments to question answering datasets . they outline key lessons that can transfer to QA research . |
| Outcome: | The proposed model is based on the best practices of trivia tournaments . the model is used to identify the best question answering teams . |
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
Improving the Robustness of Question Answering Systems to Question Paraphrasing (P19-1)
Copied to clipboard
| Challenge: | Despite advancement of question answering systems, generalizability of QA models is a topic of concern. |
| Approach: | They propose to use a neural paraphrasing model to generate multiple paraphrased questions for a given source question and a set of paraphrase suggestions to re-train the models. |
| Outcome: | The proposed approach requires no human intervention to re-train the models for improved robustness to question paraphrasing. |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
Robust Question Answering Through Sub-part Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Current textual question answering models fail to generalize to out-of-domain settings. |
| Approach: | They propose to decompose question and context into smaller units and align them to find the answer. |
| Outcome: | The proposed model is more robust than the standard BERT QA model on adversarial and out-of-domain datasets. |
Open-Domain Question Answering (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA) |
| Approach: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA . |
| Outcome: | The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods . |
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept. |
| Approach: | They propose a method for automatically building probe datasets from expert knowledge sources, allowing systematic control and a comprehensive evaluation. |
| Outcome: | The proposed model is predisposed to recognize certain types of structural linguistic knowledge, but performance degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy. |