| Challenge: | Despite substantial overlap, subtle but significant distinctions exert an outsize influence on research . one paradigm values creating more intelligent QA systems, the other paradigm values building QA system that appeals to users. |
| Approach: | They propose to use the Cranfield and Manchester paradigms to describe research working towards building human-like, intelligent QA systems. |
| Outcome: | The proposed paradigms are based on the findings of two recent studies on question answering (QA) the Cranfield paradigm is not new, but the Manchester paradigm is christened as the most eclectic in QA . |
Similar Papers
What Question Answering can Learn from Trivia Nerds (2020.acl-main)
Copied to clipboard
| Challenge: | a question answering dataset is a competition that has a leaderboard that determines the best answers. |
| Approach: | They propose to apply the best practices of trivia tournaments to question answering datasets . they outline key lessons that can transfer to QA research . |
| Outcome: | The proposed model is based on the best practices of trivia tournaments . the model is used to identify the best question answering teams . |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
Open-Domain Question Answering (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA) |
| Approach: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA . |
| Outcome: | The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods . |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
Do Question Answering Modeling Improvements Hold Across Benchmarks? (2023.acl-long)
Copied to clipboard
| Challenge: | a new study finds that human-constructed and downsampled benchmarks hold more concurrence than downsampled benchmarks. |
| Approach: | They propose to measure concurrence between two QA benchmarks on a set of 20 models . they find that human-constructed benchmarks have high concurrence amongst themselves . |
| Outcome: | The proposed models hold broadly across the diverse landscape of question answering (QA) benchmarks. |
Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (D19-58)
Copied to clipboard
| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (P19-1)
Copied to clipboard
| Challenge: | Existing question-answering systems are limited in their ability to test reasoning and comprehension. |
| Approach: | They propose a method to automatically extract implications from QA datasets to evaluate models' consistency . they use a heuristic to generate such questions and retrain models with implication-augmented data . |
| Outcome: | The proposed method shows that generated implications are well formed and valid . retraining with implication-augmented data improves consistency on both synthetic and human-generated implications. |
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing question answering datasets lack diversity in gender, profession, and nationality. |
| Approach: | They focus on how well QA models generalize across demographic subsets . english-language QA datasets mostly ask about US men from a few professions - this is problematic because most English speakers are not from the US or UK . |
| Outcome: | The proposed model accuracy is lower for people based on gender, profession, and nationality, but there is more variation on professions (question topic) and question ambiguity. |
Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods often fail to produce objective results and favor high similarity to the ground-truth question-answer pairs. |
| Approach: | They propose an alternative approach to evaluate question-answer generation using Generative Interpretation (GI) GI outperforms existing evaluation methods in terms of human alignment . |
| Outcome: | The proposed approach outperforms existing evaluation methods in human alignment and shows comparable performance with GPT3.5, only with BART-large. |