Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (D18-1)
Copied to clipboard
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme
| Challenge: | a plethora of new natural language inference datasets has been created in recent years . however, these datasets do not provide clear insight into what type of reasoning or inference a model may be performing. |
| Approach: | They propose to recast 13 existing natural language inference datasets into a common structure. |
| Outcome: | The proposed datasets provide insight into how well a sentence representation captures distinct types of reasoning. |
Similar Papers
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (N18-1)
Copied to clipboard
| Challenge: | et al., 1996, show that many of the most actively studied problems in NLP depend in large part on natural language understanding (NLU). |
| Approach: | They propose a dataset for machine learning that uses ten different genres of English to evaluate sentences for their meanings. |
| Outcome: | The multi-genre natural language inference corpus is one of the largest available for natural language understanding. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (D19-60)
Copied to clipboard
| Challenge: | Workshop on Commonsense Inference in Natural Language Processing focuses on commonsense knowledge representation and application in NLP tasks. |
| Approach: | COIN is a workshop on commonsense inference in natural language processing . workshop included two shared tasks on reading comprehension using commonsensense knowledge . |
| Outcome: | the workshop focused on modeling commonsense knowledge and commonsensing in natural language processing tasks. |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
OCNLI: Original Chinese Natural Language Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to extend natural language understanding to other languages have focused on (automatically) translating existing English datasets. |
| Approach: | They propose to use a Chinese dataset to generate annotated sentences from native speakers specializing in linguistics to elicit annotations. |
| Outcome: | The proposed dataset does not rely on automatic translation or non-expert annotation. instead, it elicits annotations from native speakers specializing in linguistics. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
A synthetic data approach for domain generalization of NLI models (2024.acl-long)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset . |
| Approach: | They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings . |
| Outcome: | The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy. |