Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
Similar Papers
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
A synthetic data approach for domain generalization of NLI models (2024.acl-long)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset . |
| Approach: | They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings . |
| Outcome: | The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy. |
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (D18-1)
Copied to clipboard
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme
| Challenge: | a plethora of new natural language inference datasets has been created in recent years . however, these datasets do not provide clear insight into what type of reasoning or inference a model may be performing. |
| Approach: | They propose to recast 13 existing natural language inference datasets into a common structure. |
| Outcome: | The proposed datasets provide insight into how well a sentence representation captures distinct types of reasoning. |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | lexical overlap heuristics are effective for frequent example types but break down in more challenging cases. |
| Approach: | They propose to use a set of examples to test whether a sentence entails another . they propose to adopt three fallible syntactic heuristics for statistical NLI models . |
| Outcome: | The proposed model performs poorly on the HANS dataset, suggesting they have adopted these heuristics. |
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (2022.aacl-main)
Copied to clipboard
| Challenge: | Negation is an important linguistic phenomenon which denotes non-existence, denial, or contradiction. |
| Approach: | They propose a natural language inference test suite to test models for negation . they use a linguistic framework to analyze negation types and constructions . |
| Outcome: | The proposed test suite is more challenging than existing benchmarks on negation . it includes annotation of negation types and constructions grounded in linguistic theory . |
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (D19-60)
Copied to clipboard
| Challenge: | Workshop on Commonsense Inference in Natural Language Processing focuses on commonsense knowledge representation and application in NLP tasks. |
| Approach: | COIN is a workshop on commonsense inference in natural language processing . workshop included two shared tasks on reading comprehension using commonsensense knowledge . |
| Outcome: | the workshop focused on modeling commonsense knowledge and commonsensing in natural language processing tasks. |