| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
Similar Papers
Using contradictions improves question answering systems (2023.acl-short)
Copied to clipboard
| Challenge: | Existing systems that use contradiction to determine if a question is supported by background contexts do better than those that use entailment. |
| Approach: | They propose a method that incorporates contradiction in natural language inference (NLI) they propose to reformulate answers from QA systems as hypotheses and then select the best one based on the results. |
| Outcome: | The proposed method improves on multiple choice and extractive QA in two settings. |
QA-NatVer: Question Answering for Natural Logic-based Fact Verification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim and its evidence via set-theoretic operators. |
| Approach: | They propose to use question answering to predict natural logic operators using generalization capabilities of instruction-tuned language models. |
| Outcome: | The proposed approach outperforms the best baseline on a Danish verification dataset by 4.3 accuracy points. |
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
Complex Reasoning in Natural Language (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Recent research shows that pretrained language models are often brittle for complex reasoning tasks. |
| Approach: | They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks . |
| Outcome: | This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness . |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work is limited in using small benchmarks with high test-train overlaps. |
| Approach: | They construct a dataset of closed-book QA using SQuAD and investigate the performance of BART. |
| Outcome: | Experiments show that pre-trained language models can achieve high performance on closed-book QA tasks. |
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in modeling and datasets demonstrate promising performance for NLI. |
| Approach: | They explore the direct zero-shot applicability of NLI models to real applications . they analyze the robustness of models to longer and out-of-domain inputs . |
| Outcome: | The proposed models are robust to longer and out-of-domain inputs and can perform on full documents. |
Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI. |
| Approach: | They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases. |
| Outcome: | The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved . |