| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
Similar Papers
Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inference (2025.coling-main)
Copied to clipboard
| Challenge: | Creating NLP datasets with Large Language Models (LLMs) is an attractive alternative to relying on crowd-source workers. |
| Approach: | They recreate a portion of the Stanford Natural Language Inference corpus using GPT-4, Llama-2 70b for Chat, and Mistral 7b Instruct. |
| Outcome: | The proposed model can be used to generate NLP datasets with stereotypical biases and annotation artifacts. |
MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain (2021.acl-short)
Copied to clipboard
| Challenge: | a large number of crowdworker-constructed datasets have been used to conduct natural language inference (NLI) on unstructured, domainspecific texts such as patient notes, pathology reports, and scientific papers. |
| Approach: | They investigate whether MedNLI contains lexical and syntactic annotation artifacts associated with annotation process that allow hypothesis-only classifiers to achieve better-than-random performance. |
| Outcome: | The proposed model outperforms a majority-class baseline model on a physician-annotated dataset with premises extracted from clinical notes. |
Evaluating BERT for natural language inference: A case study on the CommitmentBank (D19-1)
Copied to clipboard
| Challenge: | Natural language inference datasets can identify premise-hypothesis relationship without observing premise . recasting of the CommitmentBank for NLI creates hypotheses that stand in entailment/contradiction/neutral relationship with premise. |
| Approach: | They propose to recast the CommitmentBank for NLI to stand in certain relationships with the premise . hypotheses are complements of clause-embedding verbs in each premise, rethinking the CommittedBank . |
| Outcome: | The proposed model performs well on the CommitmentBank with 85% F1 . however, the model does not capture the full complexity of pragmatic reasoning, authors say . |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows . |
| Approach: | They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase . |
| Outcome: | The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods . |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
New Protocols and Negative Results for Textual Entailment Data Collection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language inference data has proven useful in benchmarking and as pretraining data for tasks requiring language understanding. |
| Approach: | They propose four alternative protocols to improve annotation quality and diversity . they use 8.5k-example training sets to compare different protocols . |
| Outcome: | The proposed protocols improve the ease of training and quality of the examples. |
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020.aacl-main)
Copied to clipboard
| Challenge: | Large-scale natural language inference datasets are available for non-expert crowdsourcing. |
| Approach: | They propose to use crowdworker writing to create candidate pairs for annotation . they find crowdworkers writing is the best known protocol for entailment data . |
| Outcome: | The proposed protocol outperforms the baseline protocol in evaluations of generalization and transfer learning. |
Baselines and Test Data for Cross-Lingual Inference (L18-1)
Copied to clipboard
| Challenge: | Recent research on textual entailment is limited to English, but it is expanding to other languages. |
| Approach: | They propose to extend the research in SNLI-style natural language inference toward multilingual evaluation by using cross-lingual word embeddings and machine translation. |
| Outcome: | The proposed system scores an average accuracy of just over 75%, but it is not perfect. |
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (D18-1)
Copied to clipboard
| Challenge: | a new dataset presents a task of grounded commonsense inference, unifying natural language inference and commonsensical reasoning. |
| Approach: | They propose a procedure that constructs a de-biased dataset by iteratively training stylistic classifiers and using them to filter the data. |
| Outcome: | The proposed procedure oversamples a de-biased dataset using state-of-the-art language models . human models struggle on the proposed procedure, indicating significant opportunities for future research. |