Papers with SNLI
Towards a Task-Agnostic Model of Difficulty Estimation for Supervised Learning Tasks (2020.aacl-srw)
Copied to clipboard
| Challenge: | Recent advances on natural language processing (NLP) benchmarks have been driven by increasingly sophisticated language models, which are pretrained on enormous amounts of data before use. |
| Approach: | They propose to develop a task-agnostic model for problem difficulty and apply it to the Stanford Natural Language Inference dataset. |
| Outcome: | The proposed model predicts how many annotators will answer a question correctly and then projectes the difficulty estimates onto the full SNLI train set to create the curriculum. |
A logical-based corpus for cross-lingual evaluation (D19-61)
Copied to clipboard
| Challenge: | a recent study shows that deep learning models can be used to solve textual inference tasks using simple linguistic patterns. |
| Approach: | They propose a set of syntactic tasks focused on contradiction detection that exploit linguistic patterns. |
| Outcome: | The proposed tasks can be implemented in English and Portuguese. |
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (D19-61)
Copied to clipboard
| Challenge: | Statistical natural language inference models are susceptible to learning dataset bias. |
| Approach: | They propose a debiasing algorithm that debiases models that use only known dataset biases . they use two benchmark datasets to train three high-performing NLI models . |
| Outcome: | The proposed learning objective improves model performance on challenge datasets while maintaining reasonable performance on original datasets. |
Annotation Artifacts in Natural Language Inference Data (N18-2)
Copied to clipboard
| Challenge: | Large-scale datasets for natural language inference are created by crowdsourcing annotations . authors show that success of natural language models to date has been overestimated . |
| Approach: | They propose a method for crowdsourcing annotations to generate 3 new sentences based on a sentence (premise) they show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI . |
| Outcome: | The proposed model can classify the hypothesis alone in 67% of SNLI and 53% of MultiNLI datasets. |
Embedding WordNet Knowledge for Textual Entailment (C18-1)
Copied to clipboard
| Challenge: | Existing deep learning models for textual entailment do not require any feature engineering or linguistic analysis. |
| Approach: | They propose to embed WordNet-derived lexical entailment relations into specially-learned word vectors and incorporate them into a decomposable attention model for textual enlightment. |
| Outcome: | The proposed model significantly improves on the SICK and SNLI datasets. |
Learning beyond Datasets: Knowledge Graph Augmented Neural Networks for Natural Language Processing (N18-1)
Copied to clipboard
| Challenge: | Currently, machine learning is limited in scalability and is limited to specific training data. |
| Approach: | They propose to enhance learning models with world knowledge in the form of Knowledge Graph fact triples for natural language processing tasks. |
| Outcome: | The proposed method is highly scalable to the amount of prior information that has to be processed and can be applied to any generic NLP task. |
Frame-Semantic Knowledge Injection for Event-Level Inference in LLMs (2026.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are fluent but often brittle when interpretation depends on external information. |
| Approach: | They propose a framework that injects frame-semantic knowledge into Large Language Models via LoRA. |
| Outcome: | The proposed framework can generalize beyond surface cues in large language models. |
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020.aacl-main)
Copied to clipboard
| Challenge: | Large-scale natural language inference datasets are available for non-expert crowdsourcing. |
| Approach: | They propose to use crowdworker writing to create candidate pairs for annotation . they find crowdworkers writing is the best known protocol for entailment data . |
| Outcome: | The proposed protocol outperforms the baseline protocol in evaluations of generalization and transfer learning. |
Benchmarking Long-tail Generalization with Likelihood Splits (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to test out of distribution generalization have their own strengths and weaknesses. |
| Approach: | They propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. |
| Outcome: | The proposed approach can be customized to construct meaningful splits for a wide range of tasks. |
Towards Improving Adversarial Training of NLP Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for generating NLP adversarial examples involve combinatorial search and expensive sentence encoders for constraining the generated instances. |
| Approach: | They propose to use vanilla adversarial training to train NLP models using a word substitution attack optimized for vanilla adversary training. |
| Outcome: | The proposed approach improves model performance and standard accuracy and can defend against other types of word substitution attacks. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Generating Token-Level Explanations for Natural Language Inference (N19-1)
Copied to clipboard
| Challenge: | Existing methods to generate token-level explanations for NLI on single sentences have not been tested. |
| Approach: | They propose to generate token-level explanations for NLI without explicitly annotating training data. |
| Outcome: | The proposed approach is faster and more accurate than the black-box methods. |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods of logical reasoning focus on entity-aware information but ignore hierarchical relations that may even have mutual effects. |
| Approach: | They propose a holistic graph network that deals with context at both discourse-level and word-level as the basis for logical reasoning. |
| Outcome: | The proposed method improves on logical reasoning QA datasets and natural language inference datasets. |
Influence Scores at Scale for Efficient Language Data Sampling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that ML models can be fine-tuned on as much data as possible without degradation in performance metrics. |
| Approach: | They evaluate the applicability of influence scores in language classification tasks by random sampling and stress-testing one of the scores. |
| Outcome: | The proposed model can be fine-tuned on 50% of the original data without degradation in performance metrics. |
Are Prompt-based Models Clueless? (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective. |
| Approach: | They propose to examine whether few-shot prompt-based models exploit superficial cues by reusing the model head and formatting the input to match the pre-training objective. |
| Outcome: | The proposed models perform well on instances with superficial cues, but often outperform random accuracy on instances without superficial cuing. |
Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness (N18-1)
Copied to clipboard
| Challenge: | Currently, state-of-the-art models achieve impressive test set performance in the form of accuracy scores. |
| Approach: | They examine the models' robustness to semantically-valid alterations to the input data by identifying three factors and comparing their impact on three SNLI models. |
| Outcome: | The proposed models show that they can generalise to new in-domain instances while also showing that they suffer from insensitivity to small but semantically significant alterations. |
Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | Using a new architecture, alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Approach: | They propose a new architecture where alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Outcome: | The proposed architecture achieves competitive performance on three popular benchmarks, SNLI, MultiNLI and SciTail, while maintaining lightweight parameter size. |
Lessons from Natural Language Inference in the Clinical Domain (D18-1)
Copied to clipboard
| Challenge: | State of the art models with deep neural networks lack generalization capabilities in specialized domains where training data is limited. |
| Approach: | They propose a dataset annotated by doctors performing a natural language inference task grounded in the medical history of patients. |
| Outcome: | The proposed model outperforms existing models in the clinical domain by incorporating domain knowledge from external data and lexical sources. |
Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets (2022.acl-long)
Copied to clipboard
| Challenge: | Natural language processing models exploit spurious correlations between features and labels in datasets to perform well only within the distributions they are trained on. |
| Approach: | They propose to generate a debiased version of a dataset and replace it with training data to train a model that is generalised to different task distributions. |
| Outcome: | The proposed method outperforms or performs comparable to state-of-the-art debiasing strategies on a large suite of debiased, out-of distribution, and adversarial test sets. |
KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches in NLP focus on “WHY A” rather than contrastive “WHA NOT B” Experimental results show that contrastive explanations are beneficial to fit the scenarios by clarifying the difference between the predicted answer and other possible wrong ones. |
| Approach: | They propose to generate contrastive explanations with counterfactual examples in NLI by identifying key phrases from input sentences and using them as key perturbations to generate counterfacts. |
| Outcome: | The proposed framework improves on SNLI and ETPA models by 91.9%. |
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
Grounded Textual Entailment (C18-1)
Copied to clipboard
Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, Albert Gatt
| Challenge: | Existing models for entailment analysis are not performing well in visual information-based models. |
| Approach: | They propose to use a visual representation of the Textual Entailment task to compare visual-grounded models with a multimodal version of the SNLI dataset. |
| Outcome: | The proposed model performs better when there is an image of the “world” or “situation” . |
Universal Adversarial Triggers for Attacking and Analyzing NLP (D19-1)
Copied to clipboard
| Challenge: | Using adversarial triggers, a model can produce a specific prediction . adversarial attacks are useful for evaluation and interpretation . |
| Approach: | They propose a gradient-guided search over tokens that finds short adversarial triggers that successfully trigger the target prediction. |
| Outcome: | The proposed algorithm finds short trigger sequences that successfully trigger the target prediction. |
Neural Natural Language Inference Models Enhanced with External Knowledge (P18-1)
Copied to clipboard
| Challenge: | Existing datasets that allow for complex models to be trained are limited . if data is not available, can machines learn all knowledge needed to perform natural language inference? |
| Approach: | They propose to enrich neural natural language inference models with external knowledge . they propose to use this knowledge to build NLI models to leverage it . |
| Outcome: | The proposed models improve on the SNLI and MultiNLI datasets. |
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples (P18-1)
Copied to clipboard
| Challenge: | Recent deep learning entailment systems have achieved close to human level performance on large datasets, but the problem is far from solved. |
| Approach: | They propose a knowledge-guided adversarial example generator for incorporating large lexical resources into entailment models via only a handful of rule templates and a natural language example generator that iteratively adjusts to the discriminator’s weaknesses. |
| Outcome: | The proposed methods increase accuracy by 4.7% on SciTail and 2.8% on a 1% sub-sample of SNLI. |
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation (N19-1)
Copied to clipboard
| Challenge: | Text generation with generative adversarial networks (GANs) can be divided into text-based and code-based categories depending on the type of signals used for discrimination. |
| Approach: | They propose a text-based approach to exploit generative adversarial networks (GANs) by using autoencoders to provide a continuous representation of sentences, which they will refer to as soft-text, and hybrid latent code and text-oriented approaches with one or more discriminators. |
| Outcome: | The proposed approach outperforms the traditional GAN-based methods on two well-known datasets. |
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (L18-1)
Copied to clipboard
| Challenge: | a method to improve the quality of training data is needed . annotation errors of dialog act corpus mislead learning results of Bayesian network . |
| Approach: | They propose to introduce a null hypothesis for predictability of textual entailment labels and test it using a Naive Bayes model. |
| Outcome: | The proposed method does not reject the null hypothesis, but it improves on the existing models. |
Explanation-based Finetuning Makes Models More Robust to Spurious Cues (2023.acl-long)
Copied to clipboard
Josh Magnus Ludan, Yixuan Meng, Tai Nguyen, Saurabh Shah, Qing Lyu, Marianna Apidianaki, Chris Callison-Burch
| Challenge: | Large Language Models (LLMs) learn correlations between labels and features that are irrelevant to the task, leading to poor generalization on out-of-distribution data. |
| Approach: | They propose an explanation-based approach to fine tune large language models to generate a free-text explanation supporting their answer. |
| Outcome: | The proposed model is more robust against spurious cues in terms of accuracy drop across four classification tasks: ComVE (+1.2), CREAK (+9.1), e-SNLI (+5.4), and SBIC (+6.5). |
Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI. |
| Approach: | They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases. |
| Outcome: | The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved . |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to train models to provide natural language explanations (NLEs) require acquisition of task-specific NLEs, which is time- and resource-consuming. |
| Approach: | They propose a few-shot out-of-domain transfer of NLEs from a parent task to a child task . they propose four methods that cover possible fine-tuning combinations of NLESs and labels . |
| Outcome: | The proposed methods cover the possible fine-tuning combinations of labels and NLEs for the parent and child tasks. |
Original Semantics-Oriented Attention and Deep Fusion Network for Sentence Matching (D19-1)
Copied to clipboard
| Challenge: | Sentence matching is a key issue in natural language inference and paraphrase identification. |
| Approach: | They propose a semantics-oriented attention and deep fusion network (OSOA-DFN) that is oriented to the original semantic representation of another sentence and propagates attention information at each matching layer. |
| Outcome: | The proposed model can model sentence matching more precisely on three sentence matching benchmark datasets. |
Asynchronous Deep Interaction Network for Natural Language Inference (D19-1)
Copied to clipboard
| Challenge: | Existing methods have framed the reasoning problem as a semantic matching task. |
| Approach: | They propose an asynchronous deep interaction network (ADIN) to deconstruct the reasoning process and implement asynchron and multi-step reasoning. |
| Outcome: | The proposed model outperforms strong baselines on three popular benchmarks: SNLI, MultiNLI, and SciTail. |
Repurposing Entailment for Multi-Hop Question Answering Tasks (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to use entailment models for question answering are limited . large scale datasets are typically framed at a sentence level, whereas question answering requires verifying whether multiple sentences, taken together as a premise, entitle a hypothesis. |
| Approach: | They propose a general architecture that can use entailment models for multi-hop QA tasks. |
| Outcome: | The proposed model outperforms QA models trained on target datasets and the OpenAI transformer models. |
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that generate NL explanations for tasks have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. |
| Approach: | They propose to use a model as a proxy for a human observer to evaluate NL explanations from the model simulatability perspective. |
| Outcome: | The proposed model-generated explanations are evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. |
LIMIT-BERT : Linguistics Informed Multi-Task BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models are usually trained on large amounts of unlabeled text data. |
| Approach: | They propose a multi-task language representations learning framework for multi-linguistics tasks by Multi-Task Learning. |
| Outcome: | The proposed model outperforms the baseline Whole Word Masking BERT on both dependency and constituent syntactic/semantic parsing, GLUE benchmark, and SNLI task. |
Certified Robustness to Adversarial Word Substitutions (D19-1)
Copied to clipboard
| Challenge: | State-of-the-art NLP models can be fooled by adversaries that apply seemingly innocuous label-preserving transformations to input text. |
| Approach: | They propose to train models that are provably robust to all word substitutions in a family of label-preserving transformations that can be replaced with a similar word without changing the original sentiment. |
| Outcome: | The proposed models achieve 75% adversarial accuracy on both sentiment analysis and natural language inference on IMDB and SNLI compared to models trained normally and ones trained with data augmentation. |
Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training. |
| Approach: | They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model. |
| Outcome: | The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement) |
Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that humans are not supervised by the natural language inference . |
| Approach: | They propose to solve the natural language inference problem via task-agnostic multimodal pretraining. |
| Outcome: | The proposed network outperforms fully-supervised BiLSTM and BiLS+ELMO on plain text inference datasets. |
Continual Few-Shot Learning for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large number of end-to-end systems are needed for many tasks in natural language processing. |
| Approach: | They propose a continual few-shot learning task where a system is asked to correct mistakes with a few training examples. |
| Outcome: | The proposed task compares two NLI and one sentiment analysis datasets with baselines from diverse paradigms. |
Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models that use only rationales to explain a prediction are limited by the complexity of deep neural networks. |
| Approach: | They extend selective rationalization to text matching by using optimal transport to find a minimal cost alignment between inputs. |
| Outcome: | The proposed model achieves very sparse rationale selections with high fidelity while preserving prediction accuracy compared to strong attention baseline models. |
Misleading Failures of Partial-input Baselines (P19-1)
Copied to clipboard
| Challenge: | Recent work establishes dataset difficulty and removes annotation artifacts via partial-input baselines. |
| Approach: | They propose to use partial-input baselines to establish dataset difficulty . they show how trivial patterns only visible in the full input can evade partial-output baseline . |
| Outcome: | The proposed model can solve 15% of previously-thought "hard" examples. |
Generating Fluent Adversarial Examples for Natural Languages (P19-1)
Copied to clipboard
| Challenge: | Current methods for building adversarial attackers for NLP are inefficient as the gradient is discarded. |
| Approach: | They propose an adversarial attacker which performs Metropolis-Hastings sampling with the guidance of gradients to solve these problems. |
| Outcome: | The proposed algorithm outperforms the baseline model on attacking capability on IMDB and SNLI. |
Knowledge-Enhanced Natural Language Inference Based on Knowledge Graphs (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to natural language inference rely on semantic knowledge, but background knowledge is limited to a few specific types. |
| Approach: | They propose a Knowledge Graph-enhanced NLI model that leverages background knowledge stored in knowledge graphs to facilitate inference. |
| Outcome: | The proposed model can leverage background knowledge stored in knowledge graphs to perform the task. |
Baselines and Test Data for Cross-Lingual Inference (L18-1)
Copied to clipboard
| Challenge: | Recent research on textual entailment is limited to English, but it is expanding to other languages. |
| Approach: | They propose to extend the research in SNLI-style natural language inference toward multilingual evaluation by using cross-lingual word embeddings and machine translation. |
| Outcome: | The proposed system scores an average accuracy of just over 75%, but it is not perfect. |
Detecting Label Errors by Using Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label error detection focus on label errors in training data. |
| Approach: | They propose a method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. |
| Outcome: | The proposed method outperforms existing methods for detecting label errors in natural language datasets. |
Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching (D19-1)
Copied to clipboard
| Challenge: | Recent work on text sequence matching tasks uses task specific supervised datasets, which are always limited to the amount due to the cost of annotation. |
| Approach: | They propose an aggregation method to combine Bidirectional Encoder Representations from Transformer (BERT) with a MatchLSTM layer for Sequence Matching. |
| Outcome: | The proposed model improves on two publicly available datasets, WikiQA and SNLI. |
GrASP: A Library for Extracting and Exploring Human-Interpretable Textual Patterns (2022.lrec-1)
Copied to clipboard
| Challenge: | a Python library is available for extracting patterns from textual data. |
| Approach: | They propose a Python library for extracting patterns from textual data . it integrates a public implementation of the existing GrASP algorithm . |
| Outcome: | The proposed library integrates a public implementation of the existing GrASP algorithm. |
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
Uncertain Natural Language Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Existing categorical bin assignments for NLI are not effective for entailment inference. |
| Approach: | They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments. |
| Outcome: | The proposed model can be used in pre-training and correlates well with humans. |
Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations (2023.acl-long)
Copied to clipboard
| Challenge: | Human-annotated labels and explanations are critical for training explainable NLP models. |
| Approach: | They propose a metric that measures the usefulness of an explanation for model performance at both fine-tuning and inference. |
| Outcome: | The proposed metric can evaluate the quality of human-annotated explanations, while Simulatability falls short. |
Automated Fact-Checking of Claims from Wikipedia (2020.lrec-1)
Copied to clipboard
| Challenge: | Fact checking datasets such as FEVER and SNLI suffer from limited applicability due to synthetic nature of claims and/or evidence written by annotators that differ from real claims and evidence on the internet. |
| Approach: | They present a dataset of 124k+ triples consisting of a claim, context and an evidence document extracted from English Wikipedia articles and citations. |
| Outcome: | The proposed dataset is the largest fact checking dataset consisting of real claims and evidence to date. |
Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that the interaction of large language models (LLMs) with theorem provers (TPs) can help verify and improve the validity of NLI explanations. |
| Approach: | They propose to use logical expressions to guide LLMs in generating structured proof sketches and to use them to improve their accuracy. |
| Outcome: | The proposed strategies improve autoformalisation, syntactic errors and explanation refinement over the state-of-the-art model. |
Rule Discovery for Natural Language Inference Data Generation Using Out-of-Distribution Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training rules for natural language inference do not cover the diversity of natural language. |
| Approach: | They propose a framework that combines out-of-distribution detection and clustering to identify new premise–hypothesis pairs in a dataset that are not covered by existing rules. |
| Outcome: | The proposed framework achieves +0.85%p accuracy on 2k and +0.15%p on 550k samples. |
To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)
Copied to clipboard
| Challenge: | State-of-the-art language models perform well on a variety of language tasks, but struggle with understanding negation cues in tasks like natural language inference (NLI). |
| Approach: | They propose a new learning strategy for negation building on ELECTRA’s replaced token detection objective. |
| Outcome: | The proposed approach leads to substantial gains on a variant of RTE with additional negation. |
Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional methods that rely on 1:1 pairwise comparisons fail to capture inconsistencies . few studies have addressed related problems in the domain of factual inconsistenency detection . |
| Approach: | They propose a set-consistency verification task that assesses logical coherence of entire sets . they propose 'set-consistent energy network' that employs a margin-based loss to learn the compatibility among a collection of statements . |
| Outcome: | The proposed model outperforms existing methods and significantly outperformed existing models. |
LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge. |
| Approach: | They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation. |
| Outcome: | The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies. |