Papers with NLI
Improving Medical NLI Using Context-Aware Domain Knowledge (2020.starsem-1)
Copied to clipboard
| Challenge: | Domain knowledge is important to understand both the lexical and relational associations of words in natural language text . lack of annotated dataset can lead to wrong inference predictions . |
| Approach: | They propose a knowledge adaptive approach that encodes the premise/hypothesis texts by leveraging supplementary external knowledge alongside the UMLS based on the word contexts. |
| Outcome: | The proposed model can align token-level interactions between the premise and hypothesis more effectively. |
Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the application and evaluation of Large Language Models (LLMs) but LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains. |
| Approach: | They propose an abductive-deductive framework that integrates Large Language Models with an external backward-chaining solver to refine step-wise natural language explanations. |
| Outcome: | The proposed framework improves explanations generated via in-context learning methods and Chain-of-Thought (CoT) on ethical NLI tasks while producing formal proofs describing and supporting models’ reasoning. |
Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing solutions for hallucination detection do not consider latency, train or evaluate on production data. |
| Approach: | They propose to use customer service conversation data to evaluate existing methods . they propose to train small encoder models on a new dataset to outperform existing methods. |
| Outcome: | The proposed model outperforms existing methods and highlights the value of combining small amounts of in-domain data with public datasets. |
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |
Reading Comprehension as Natural Language Inference:A Semantic Analysis (2020.starsem-1)
Copied to clipboard
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Li, Pavan Kapanipathi, Kartik Talamadupula
| Challenge: | In recent past, Natural language Inference (NLI) has gained significant attention, but its true impact has not been well studied. |
| Approach: | They propose to transform a large RACE dataset into an NLI model and compare it to a state-of-the-art model. |
| Outcome: | The proposed model outperforms the previous model on a question-answer concatenation form and a coherent entailment form. |
Have Attention Heads in BERT Learned Constituency Grammar? (2021.eacl-srw)
Copied to clipboard
| Challenge: | Recent pre-trained language models have gained great success in many tasks, but what they have learned, and when they perform well remain unknown. |
| Approach: | They employ the syntactic distance method to extract implicit constituency grammar from attention weights of attention heads of BERT and RoBERTa. |
| Outcome: | The proposed models induce some grammar types much better than baselines, suggesting some heads act as a proxy for constituency grammar. |
New Frontiers of Information Extraction (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Information extraction (IE) is the process of automatically extracting structural information from unstructured or semi-structured data. |
| Approach: | This tutorial will provide an introduction to recent advances in IE by answering several important research questions. |
| Outcome: | The tutorial will address several important research questions and outline directions for further investigation. |
When Truth Matters - Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs) (2023.starsem-1)
Copied to clipboard
| Challenge: | In this paper, we examine the ability of large language models (LLMs) to accommodate different pragmatic sentence types, such as questions, commands, and sentence fragments for natural language inference (NLI). |
| Approach: | They propose to fine-tune large language models to accommodate different sentence types for natural language inference (NLI) they also explore ChatGPT's concept of entailment by using a symbolic semantic parser. |
| Outcome: | The proposed models can accommodate different sentence types without losing too much accuracy on MNLI-matched models. |
Semantic Diversity in Dialogue with Natural Language Inference (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to measure diversity of chitchat model responses have been proposed to measure iteratively. |
| Approach: | They propose a metric which uses Natural Language Inference to measure the semantic diversity of a set of model responses for a conversation. |
| Outcome: | The proposed metric improves the diversity of a sampled set of responses using a new generation procedure called Diversity Threshold Generation. |
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (D18-1)
Copied to clipboard
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme
| Challenge: | a plethora of new natural language inference datasets has been created in recent years . however, these datasets do not provide clear insight into what type of reasoning or inference a model may be performing. |
| Approach: | They propose to recast 13 existing natural language inference datasets into a common structure. |
| Outcome: | The proposed datasets provide insight into how well a sentence representation captures distinct types of reasoning. |
NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning (2021.starsem-1)
Copied to clipboard
| Challenge: | Currently, symbolic and deep learning approaches to NLI are receiving less attention. |
| Approach: | They propose a symbolic-based inference framework that integrates symbolic reasoning and semantic formalism to solve NLI tasks. |
| Outcome: | The proposed framework improves accuracy on the NLI task and on the SICK and MED datasets. |
Logical Inference for Counting on Semi-structured Tables (2022.acl-srw)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) tasks require numerical understanding to perform a numerical type of inference, such as counting. |
| Approach: | They propose a logical inference system for reasoning between semi-structured tables and texts that uses logical representations as meaning representations and model checking to handle a numerical type of inference. |
| Outcome: | The proposed system can perform inference with numerical comparatives with tables and texts in English. |
Unsupervised Question Answering for Fact-Checking (D19-66)
Copied to clipboard
| Challenge: | Recent Deep Learning (DL) models have achieved human-level accuracy on natural language tasks such as question-answering, natural language inference, and textual entailment. |
| Approach: | They propose an unsupervised question-answering based approach for a similar task, fact-checking. |
| Outcome: | The proposed approach achieves label accuracy of 80.2% on the development set and 80.25% on the test set. |
Using Classifier Features to Determine Language Transfer on Morphemes (N18-4)
Copied to clipboard
| Challenge: | Using native English data, we identify an English learner’s native language background based solely on the learner's English writing samples. |
| Approach: | They perform a Native Language Identification task where they identify an English learner’s native language background based only on the learner's English writing samples. |
| Outcome: | The proposed task is connected to a position in second language acquisition research that holds all learners acquire English grammatical morphemes in the same order, regardless of native language background. |
XplaiNLI: Explainable Natural Language Inference through Visual Analytics (2020.coling-demos)
Copied to clipboard
| Challenge: | Recent research has revealed some heuristics and biases of natural language inference models. |
| Approach: | They propose an interactive visualization interface that computes Natural Language Inference with different methods and provides explanations for the decisions made by each approach. |
| Outcome: | The proposed interface computes Natural Language Inference (NLI) with three different approaches and provides explanations for the decisions made by each approach. |
Compositional Semantics and Inference System for Temporal Order based on Japanese CCG (2022.acl-srw)
Copied to clipboard
| Challenge: | a system for temporal order in Japanese has not been developed for linguistic inference involving temporal expressions. |
| Approach: | They propose a Japanese NLI system that considers temporal order in Japanese . they use axioms for temporal relations and automated theorem provers to perform inference involving temporal orders. |
| Outcome: | The proposed system outperforms logic-based systems and current deep learning models on Japanese datasets. |
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that even state-of-the-art pre-trained language models can generate inconsistent summaries in more than 70% of all cases. |
| Approach: | They propose a method that enables NLI models to be used for inconsistency detection by segmenting documents into sentence units and aggregating scores between pairs of sentences. |
| Outcome: | The proposed method achieves state-of-the-art accuracy of 74.4% on six large inconsistency detection datasets. |
Paraphrase Identification via Textual Inference (2024.starsem-1)
Copied to clipboard
| Challenge: | Paraphrase identification (PI) and natural language inference (NLI) are important tasks in natural language processing. |
| Approach: | They propose a method for paraphrase identification and natural language inference using an NLI system to solve these tasks. |
| Outcome: | The proposed method outperforms dedicated PI models on PI datasets and provides insights into limitations of current benchmarks. |
Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning (2026.acl-srw)
Copied to clipboard
| Challenge: | Annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity, but most fine-tuning recipes treat every example as equallylearnable. |
| Approach: | They ask whether annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity. |
| Outcome: | The proposed method predicts learning dynamics on contested examples under LoRA. |
Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI). |
| Approach: | They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles. |
| Outcome: | The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles. |
A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus (2024.acl-long)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is an actively studied topic serving as a proxy for natural language understanding. |
| Approach: | They propose to use a Romanian NLI corpus to analyze sentence pairs . they use multiple machine learning methods to establish competitive baselines . |
| Outcome: | The proposed model improves on the best model by employing a new curriculum learning strategy based on data cartography. |
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (D19-61)
Copied to clipboard
| Challenge: | Statistical natural language inference models are susceptible to learning dataset bias. |
| Approach: | They propose a debiasing algorithm that debiases models that use only known dataset biases . they use two benchmark datasets to train three high-performing NLI models . |
| Outcome: | The proposed learning objective improves model performance on challenge datasets while maintaining reasonable performance on original datasets. |
A Generative Approach for Mitigating Structural Biases in Natural Language Inference (2022.starsem-1)
Copied to clipboard
| Challenge: | Natural language inference datasets contain artifacts and biases that allow models to perform poorly by using a biased subset of the input without considering the remainder features. |
| Approach: | They reformulate a natural language inference task as a generative task . they find that this approach is highly robust to large amounts of bias . |
| Outcome: | The proposed model is highly robust to large amounts of bias. |
Analyzing Dynamic Adversarial Training Data in the Limit (2022.findings-acl)
Copied to clipboard
| Challenge: | Dynamic adversarial data collection (DADC) can be used to build models that are robust across a wide range of test inputs. |
| Approach: | They propose to run Dynamic adversarial data collection over many rounds to maximize its training-time benefits. |
| Outcome: | The proposed model makes 26% fewer errors on the premise paragraphs compared to models trained on non-adversarial examples. |
Gradient-based Analysis of NLP Models is Manipulable (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that explanation techniques can be unstable and can be manipulated to hide the actual reasoning behind the predictions of NLP models. |
| Approach: | They propose to merge a BERT-based sentiment classifier with a Facade Model that overwhelms the gradients without affecting the predictions. |
| Outcome: | The proposed model overwhelms the gradients without affecting the predictions on a variety of NLP tasks, such as sentiment analysis, NLI, and QA. |
Logographic Information Aids Learning Better Representations for Natural Language Inference (2022.findings-aacl)
Copied to clipboard
| Challenge: | Statistical language models often ignore information related to logographic features of written text . many languages, including Chinese and Vietnamese, use logographic writing systems . |
| Approach: | They propose to provide language models with logographic information to learn better semantic representations. |
| Outcome: | The proposed model improves semantic representations in six languages with logographic writing systems. |
On the Role of Summary Content Units in Text Summarization Evaluation (2024.naacl-short)
Copied to clipboard
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo Walenta, Juri Opitz, Leonardo Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou
| Challenge: | a human written summary content unit (SCU) is used to judge the quality of a summary . a pyramid evaluation method is based on SCUs that decompose a reference summary into concise sentences . |
| Approach: | They propose to use automated SCUs to evaluate the quality of a candidate summary . they propose to generate SCU approximations from AMR meaning representations and large language models . |
| Outcome: | The proposed method can be fully automated, but lacks the human effort to validate it. |
Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have brought significant improvements to many natural language processing tasks. |
| Approach: | They propose a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network. |
| Outcome: | The proposed method improves on several NLI datasets using a strong baseline model (RoBERTa-large) with MNLI pretraining. |
Discovering Better Model Architectures for Medical Query Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Neural architecture search (NAS) has attracted intense attention in computer vision and NLP. |
| Approach: | They propose to use neural architecture search to optimize model architectures for medical questions . they propose to modify the ENAS method to accelerate and stabilize the search results . |
| Outcome: | The proposed approach outperforms baseline models on two medical questions . it is compared with other NAS methods and shows that it provides the best results . |
Logical Inferences with Comparatives and Generalized Quantifiers (2020.acl-srw)
Copied to clipboard
| Challenge: | Comparative constructions pose a challenge in Natural Language Inference (NLI) Comparatives interact with quantifiers, numerals, and lexical antonyms, but a logical inference system for comparatives has not been developed for the task. |
| Approach: | They propose a compositional semantics system that maps comparative constructions to semantic representations via combinatory categorial grammar parsers and integrates it with an automated theorem proving system. |
| Outcome: | The proposed system outperforms previous logic-based systems and deep learning models on three NLI datasets. |
Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for ultra-fine entity typing fail to capture type semantics because of the large number of types and the scarcity of data per type. |
| Approach: | They propose a method that formulates entity typing as a natural language inference problem . they use indirect supervision from NLI to infer type information as textual hypotheses . |
| Outcome: | The proposed method achieves state-of-the-art performance on the ultra-fine entity typing task with limited training data. |
Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (N19-1)
Copied to clipboard
| Challenge: | Sentiment analysis (SA) is a computational task that aims to identify opinion polarity towards a specific aspect. |
| Approach: | They propose to convert ABSA into a sentence-pair classification task such as question answering and natural language inference. |
| Outcome: | The proposed model is fine-tuned and achieves state-of-the-art on SentiHood and SemEval-2014 datasets. |
Sentence Representations via Gaussian Embedding (2024.eacl-short)
Copied to clipboard
| Challenge: | Sentence embeddings represent a sentence's meaning as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations. |
| Approach: | They propose a Gaussian-distribution-based contrastive learning framework for sentence embedding that can handle asymmetric inter-sentential relations and a similarity measure for identifying entailment relations. |
| Outcome: | The proposed framework performs comparable to that of previous methods on natural language inference tasks and estimates direction of entailment relations, which is difficult with point representations. |
A Question-Answer Driven Approach to Reveal Affirmative Interpretations from Verbal Negations (2022.findings-naacl)
Copied to clipboard
Md Mosharaf Hossain, Luke Holman, Anusha Kakileti, Tiffany Kao, Nathan Brito, Aaron Mathews, Eduardo Blanco
| Challenge: | Negations carry affirmative meanings, which are difficult to process and understand by humans. |
| Approach: | They propose a question-answer driven approach to reveal affirmative interpretations from verbal negations. |
| Outcome: | The proposed approach is based on a natural language inference task . it shows that state-of-the-art transformers are insufficient to reveal affirmative interpretations . |
SWING: Balancing Coverage and Faithfulness for Dialogue Summarization (2023.findings-eacl)
Copied to clipboard
Kung-Hsiang Huang, Siffi Singh, Xiaofei Ma, Wei Xiao, Feng Nan, Nicholas Dingwall, William Yang Wang, Kathleen McKeown
| Challenge: | Existing approaches to dialogue summarization rely on features of conversation data. |
| Approach: | They propose to use natural language inference models to improve coverage and faithfulness . they use fine-grained training signals to encourage model to generate missing content . |
| Outcome: | The proposed model achieves higher faithfulness and coverage while maintaining conciseness compared to prior methods. |
KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmark datasets for natural language inference and semantic textual similarity (STS) are not available in the Korean language. |
| Approach: | They construct and release new datasets for Korean NLI and STS . they machine-translate existing English training sets and manually translate development and test sets into Korean to accelerate research on Korean NLU. |
| Outcome: | The proposed datasets are available at https://github.com/kakaobrain/KorNLUDatasets. |
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023.acl-short)
Copied to clipboard
| Challenge: | Zero-shot cross-lingual transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter within a source language. |
| Approach: | They propose to use unlabeled text to enhance zero-shot transfer by pairing language adapters with task adapters in a target language. |
| Outcome: | The proposed framework improves on three cross-lingual tasks with up to 11% relative improvement in Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI). |
A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing ILP frameworks are non-differentiable and cannot be integrated as part of a broader deep learning architecture. |
| Approach: | They propose a neuro-symbolic architecture for explanation-based NLI based on DBCS. |
| Outcome: | The proposed approach achieves superior performance when compared to existing solvers and black-box solver. |
An Analysis under a Unified Formulation of Learning Algorithms with Output Constraints (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing work on NN models with output constraints has not been able to categorize them in a unified manner. |
| Approach: | They propose new algorithms to integrate the information of main task and constraint injection . they use the H-score as a metric for considering main task metric and constrain infringement simultaneously . |
| Outcome: | The proposed algorithms integrate the information of main task and constraint injection, inspired by continual-learning algorithms. |
Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples (2024.acl-srw)
Copied to clipboard
| Challenge: | Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. |
| Approach: | They propose to automatically generate an NLI dataset with an LLM and use it for fine-tuning of PromptEOL. |
| Outcome: | The proposed model outperforms existing models on STS tasks without large manually annotated datasets. |
Which Spurious Correlations Impact Reasoning in NLI Models? A Visual Interactive Diagnosis through Data-Constrained Counterfactuals (2023.acl-demo)
Copied to clipboard
| Challenge: | a spurious correlation exists when a feature correlates with the target label while there is no causal relationship between the feature and the label. |
| Approach: | They propose a dashboard that allows users to generate diverse and challenging examples by drawing inspiration from GPT-3 suggestions. |
| Outcome: | The proposed dashboard enables users to generate diverse and challenging examples by drawing inspiration from GPT-3 suggestions and make refinements based on the feedback. |
Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Intent discovery remains a crucial task in natural language processing . identifying novel, unseen intents remains one of the biggest challenges in this field . |
| Approach: | They propose a multi-language approach to intent discovery using Adapters and a Transformer architecture. |
| Outcome: | The proposed pipeline outperforms baselines in two zero-shot settings for intent classification and unseen intent discovery. |
How Fast can BERT Learn Simple Natural Language Inference? (2021.eacl-main)
Copied to clipboard
| Challenge: | Efficiency of learning of BERT is very slow due to hidden dataset bias . however, some studies show that it can learn with surface clues/patterns . |
| Approach: | They propose to use a simple entailment judgment case to test whether BERT can learn without hidden dataset bias. |
| Outcome: | The proposed case shows that BERT can learn without hidden bias without utilizing dataset bias. |
DefSent: Sentence Embeddings using Definition Sentences (2021.acl-short)
Copied to clipboard
| Challenge: | Sentence embedding methods using natural language inference datasets are limited for limited languages due to large datasets. |
| Approach: | They propose a sentence embedding method that uses definition sentences from a word dictionary. |
| Outcome: | The proposed method performs comparably on unsupervised semantics textual similarity tasks and slightly better on SentEval tasks than methods using large NLI datasets. |
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021.naacl-main)
Copied to clipboard
| Challenge: | Typical fact verification models use retrieved written evidence to verify claims . evidence sources change over time as more information is gathered and revised . a new benchmark for fact verification is VitaminC, which is contrastive in nature . |
| Approach: | They propose a benchmark that uses Wikipedia revisions to train models to discern and adjust to slight factual changes. |
| Outcome: | The proposed model improves accuracy by 10% on adversarial fact verification and 6% on adversary natural language inference. |
Leveraging Moment Injection for Enhanced Semi-supervised Natural Language Inference with Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have used class-specific fine-tuned large language models to generate hypotheses and assign pseudo-labels but discarded many LLM-constructed samples to ensure the quality. |
| Approach: | They propose to leverage LLM-constructed samples by injecting the moments of labeled samples during training to properly adjust the level of noise. |
| Outcome: | The proposed method outperforms strong baselines on multiple NLI datasets in low-resource settings. |
Developmental Negation Processing in Transformer Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012). |
| Approach: | They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories. |
| Outcome: | The proposed models perform better on certain categories, suggesting clear differences in how they are processed. |
PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are prone to factually inconsistent statements, known as hallucinations. |
| Approach: | They propose to train a specialized model that detects inconsistencies over text prefixes to improve generation faithfulness by 5-14 F1 points. |
| Outcome: | The proposed model outperforms baseline models by 5-14 F1 points in prefix-level entailment. |
Language Models Are Poor Learners of Directional Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets fail to test directionality and are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. |
| Approach: | They propose a benchmark for directional predicate entailments that is extrinsic to existing training sets. |
| Outcome: | The proposed model is incompetent on directional predicate entailments, compared to engorgement graphs, but limited by sparsity. |
TruthSplit: Revealing Conditional Validity in Arguments Through Multi-Worldview Comparative Reasoning (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing argumentation tools analyze properties of the argument, but leave perspective-specific background knowledge implicit. |
| Approach: | They propose a system that analyzes arguments from multiple perspectives using a perspective-dependent analysis approach. |
| Outcome: | The proposed system analyzes arguments from multiple perspectives and generates interpretations, value conflicts and assumption gaps. |
Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for improving consistency often train with additional NLI labels or attach trained extra modules to the generative agent. |
| Approach: | They propose to encode personas into dialogue embeddings and a persona-conditioned dialogue dataset to improve persona consistency. |
| Outcome: | The proposed approach can enforce dialogue agents to refrain from contradictions and improve consistency of existing models. |
Rethinking STS and NLI in Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. |
| Approach: | They propose to rethink semantic textual similarity (STS) and natural language inference (NLI) models with task-specific prompts and model overconfidence to capture disagreements between human judgements. |
| Outcome: | The proposed models are able to capture human opinions on individual examples without any parameter modifications. |
Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (2022.aacl-main)
Copied to clipboard
| Challenge: | Negation is an important linguistic phenomenon which denotes non-existence, denial, or contradiction. |
| Approach: | They propose a natural language inference test suite to test models for negation . they use a linguistic framework to analyze negation types and constructions . |
| Outcome: | The proposed test suite is more challenging than existing benchmarks on negation . it includes annotation of negation types and constructions grounded in linguistic theory . |
Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences (2022.coling-1)
Copied to clipboard
| Challenge: | Presuppositions are assumptions that are taken for granted by an utterance. |
| Approach: | They propose to use heuristics to create alternative "contrastive" test cases . they also analyze samples from ImpPres datasets to better understand their predictions . |
| Outcome: | The proposed model performs better on the ImpPres dataset than on the other datasets. |
EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI (2026.findings-acl)
Copied to clipboard
| Challenge: | Human label variation (HLV) arises when multiple labels are valid for the same instance. |
| Approach: | They propose a framework for generating and validating explanations to detect errors using large language models (LLMs) EVADE framework provides broader explanation coverage and requires less human intervention . |
| Outcome: | The proposed framework provides broader explanation coverage, requires less human intervention, and delivers better downstream performance in predicting label distributions. |
Too much of product information : Don’t worry, let’s look for evidence! (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing product question answering models do not provide labelled data for the task and description information for products is very lengthy. |
| Approach: | They propose a distant supervision-based NLI model to prepare training data without manual efforts. |
| Outcome: | The proposed model outperforms standard multi-task fine-tuning and improves 6% in human evaluation over baselines. |
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020.aacl-main)
Copied to clipboard
| Challenge: | Large-scale natural language inference datasets are available for non-expert crowdsourcing. |
| Approach: | They propose to use crowdworker writing to create candidate pairs for annotation . they find crowdworkers writing is the best known protocol for entailment data . |
| Outcome: | The proposed protocol outperforms the baseline protocol in evaluations of generalization and transfer learning. |
Two-Step Classification using Recasted Data for Low Resource Settings (2020.aacl-main)
Copied to clipboard
Shagun Uppal, Vivek Gupta, Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, Amanda Stent
| Challenge: | Existing studies on NLP models focus on high resource languages like English, but there are only two datasets for Hindi. |
| Approach: | They propose a novel two-step classification method which uses textual-entailment predictions for classification task. |
| Outcome: | The proposed method improves classification performance by using a joint-objective for classification and textual entailment. |
Developing a Reliable, Fast, General-Purpose Hallucination Detection and Mitigation Service (2025.naacl-industry)
Copied to clipboard
| Challenge: | Hallucination is a problem in large language models that produce incorrect output . authors propose a reliable and high-speed production system to detect and rectify hallucinations . |
| Approach: | They propose a high-speed production system that detects hallucinations in LLMs . they propose NER, natural language inference, span-based detection and a rewriting mechanism . |
| Outcome: | The proposed system detects a wide range of hallucinations in LLM responses. |
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Current LLMs are primarily trained on English data but also include data from other languages. |
| Approach: | They propose to use a pre-translation strategy to translate a task prompt into English before inference . they use 'a modular entity' that could be translated into four different languages . |
| Outcome: | The proposed strategies are based on a set of pre-trained data across 35 languages covering both low and high-resource languages. |
DocTime: A Document-level Temporal Dependency Graph Parser (2022.naacl-main)
Copied to clipboard
Puneet Mathur, Vlad Morariu, Verena Kaynig-Fittkau, Jiuxiang Gu, Franck Dernoncourt, Quan Tran, Ani Nenkova, Dinesh Manocha, Rajiv Jain
| Challenge: | Document dependency graphs (TDGs) are used to understand the temporal relations between events mentioned in a document and to improve downstream tasks such as timeline creation and time-aware summarization. |
| Approach: | They propose a temporal dependency graph parser that takes input from a text document and produces a graph that incorporates longer range dependencies. |
| Outcome: | The proposed framework outperforms existing models on three datasets and improves tasks such as timeline creation, time-aware summarization, and temporal information extraction. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
Batch Prompting: Efficient Inference with Large Language Model APIs (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Performing inference on large volumes of samples can be computationally and financially costly. |
| Approach: | They propose a prompting approach that enables large language models to run inference in batches instead of one sample at a time. |
| Outcome: | The proposed prompting reduces both token and time costs while retaining downstream performance. |
Weakly supervised hierarchical multi-task classification of customer questions (2023.acl-industry)
Copied to clipboard
Jitenkumar Rana, Promod Yenigalla, Chetan Aggarwal, Sandeep Sricharan Mukku, Manan Soni, Rashmi Patange
| Challenge: | Identifying granular and actionable topics from customer questions helps improve the overall customer experience. |
| Approach: | They propose a weakly supervised Hierarchical Multi-task Classification Framework to identify granular topics from customer questions . a clustering based taxonomy creation and data labeling module is used to create taxonomies and labelled data with minimal supervision. |
| Outcome: | The proposed model achieves 13% better accuracy over single-task classification frameworks . it can adapt to constantly evolving taxonomy without need of re-training . |
To be or not to be an Integer? Encoding Variables for Mathematical Text (2022.findings-acl)
Copied to clipboard
| Challenge: | a number of natural language inference models are limited in interpreting mathematical knowledge written in Natural Language . a variable's meaning is determined exclusively by its defining type, i.e., its context . |
| Approach: | They propose a method that can create context-based representations for variables . they propose 'variable slot' approach which can be used to model variables based on their meaning . |
| Outcome: | The proposed model can be used to represent variables in natural language . it can be applied to a task of variable typing and create context-based representations for variables . |
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (D19-1)
Copied to clipboard
| Challenge: | Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks. |
| Approach: | They propose to use a model that is pretrained on 104 languages for cross-lingual transfer. |
| Outcome: | The proposed model performs well on 5 NLP tasks covering 39 languages from various language families. |
Investigating Reasons for Disagreement in Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Several disagreements in natural language inference (NLI) annotation are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. |
| Approach: | They propose a 4-way classification approach and a multilabel classification approach for detecting disagreements in natural language inference annotations. |
| Outcome: | The proposed model is more expressive and gives better recall of possible interpretations in the data. |
Exploring Transitivity in Neural NLI Models through Veridicality (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite recent success of deep neural networks in natural language processing, the extent to which they can demonstrate human-like generalization capacities remains unclear. |
| Approach: | They propose an analysis method to evaluate whether models can draw inferences composed of veridical inference and arbitrary inference types. |
| Outcome: | The proposed model performs poorly on transitivity inference tasks, suggesting it lacks generalization capacity for drawing composite inferences from training examples. |
With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness (2023.acl-short)
Copied to clipboard
| Challenge: | Conditional language models generate unfaithful output that is not supported by their input . this jeopardizes trust in real-world applications, raising a need for automatic faithfulness metrics. |
| Approach: | They propose to augment conditional language models with robust inference procedures to improve faithfulness. |
| Outcome: | The proposed approach outperforms existing models on the TRUE benchmark. |
On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference (N18-2)
Copied to clipboard
| Challenge: | Existing methods to investigate whether sentence representations from NMT systems capture distinct semantic phenomena are limited. |
| Approach: | They propose a process to investigate the extent to which sentence representations arising from neural machine translation systems encode distinct semantic phenomena. |
| Outcome: | The proposed model is suited to supporting inferences at the syntax-semantics interface, compared to anaphora resolution requiring world knowledge. |
Elastic weight consolidation for better bias inoculation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that the lack of suitable inductive biases in sentence-pair classification models can cause misclassifications on training datasets. |
| Approach: | They propose to use elastic weight consolidation (EWC) to fine-tune models to mitigate biases while being less susceptible to catastrophic forgetting. |
| Outcome: | The proposed model improves on fact verification and stress tests while maintaining the original task accuracy. |
Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows . |
| Approach: | They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase . |
| Outcome: | The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods . |
Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Cross-lingual transfer of language models trained on high-resource languages such as English has been limited due to the high cost of obtaining non-English conversational data. |
| Approach: | They introduce a parallel and large-scale multilingual conversation dataset that is used for cross-lingual alignment pretraining by translating the English-only Schema-Guided Dialogue dataset into 105 other languages. |
| Outcome: | The proposed model performs well on slot-filling and intent classification tasks, and is able to perform well in other languages. |
SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | We evaluate a large number of strong baselines on SherLIiC, ranging from semantic vector space models to state of the art neural models of natural language inference (NLI). |
| Approach: | They propose a testbed for lexical inference in context consisting of 3985 manually annotated inference rule candidates and a set of 960k unlabeled InfCands. |
| Outcome: | The proposed testbed is based on 3985 manually annotated inference rule candidates (InfCands) and 190k typed textual relations between Freebase entities extracted from the large entity-linked corpus ClueWeb09. |
MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference (2024.naacl-long)
Copied to clipboard
| Challenge: | a dataset containing 132,320 sentence pairs from five new scientific domains is used for scientific Natural Language Inference (NLI) the availability of multiple domains makes it possible to study domain shift for scientific NLI. |
| Approach: | They propose a dataset with 132,320 sentence pairs from five new scientific domains to introduce diversity in scientific NLI. |
| Outcome: | The proposed dataset contains 132,320 sentence pairs extracted from five new scientific domains. |
Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to natural language inference focus on interaction architectures of sentences . but, we propose to transfer knowledge from discourse markers to augment the model . |
| Approach: | They propose to transfer knowledge from discourse markers to augment the quality of the NLI model. |
| Outcome: | The proposed method achieves state-of-the-art performance on large-scale datasets. |
Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate consistency capacity of open-domain chatbots are costly and low-efficient. |
| Approach: | They propose an efficient framework for evaluating consistency of open-domain chatbots . they use human judges to interact with chatbot, which is costly and low-efficient . |
| Outcome: | The proposed framework can assess the consistency capacity of chatbots and achieve a high ranking correlation with the human evaluation. |
SharPT: Shared Latent Space Prompt Tuning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prompt tuning is an efficient method for adapting large language models, but it is difficult and expensive to identify the source task that provides optimal prompts. |
| Approach: | They propose to learn a shared latent space which captures a set of basis skills from a mixture of source tasks and then transfer them to target tasks. |
| Outcome: | The proposed method outperforms previous methods on NLI, sentence completion, QA, conference resolution, word sense disambiguation and on various model scales. |
Explainable Hallucination through Natural Language Inference Mapping (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate hallucinated content, making it crucial to identify and quantify inconsistencies in their outputs. |
| Approach: | They propose a framework that maps entailment and contradiction relations between inputs and outputs using a natural language inference model. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by five percentage points while providing clear, interpretable explanations. |
Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for fact checking RAG outputs rely on large language models. |
| Approach: | They propose a method that computes a factuality score that can be thresholded to yield a binary decision to check RAG outputs. |
| Outcome: | The proposed method is low latency and low cost at run-time and no need for LLM fine-tuning. |
Generating Token-Level Explanations for Natural Language Inference (N19-1)
Copied to clipboard
| Challenge: | Existing methods to generate token-level explanations for NLI on single sentences have not been tested. |
| Approach: | They propose to generate token-level explanations for NLI without explicitly annotating training data. |
| Outcome: | The proposed approach is faster and more accurate than the black-box methods. |
A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of attribution methods focus on the English language . plausibility and faithfulness are two main criteria for plausible and faithful attributions . |
| Approach: | They propose a cross-lingual strategy to measure faithfulness based on word alignments. |
| Outcome: | The proposed approach eliminates drawbacks of erasure-based evaluations and provides a multilingual dataset with highlights to support future studies. |
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses. |
| Approach: | They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses. |
| Outcome: | The proposed model performs better on diverse summarisation tasks than existing models. |
Understanding by Understanding Not: Modeling Negation in Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Negation is a core construction in natural language, but state-of-the-art pre-trained language models often handle it incorrectly. |
| Approach: | They propose to augment language modeling objective with unlikelihood objective based on negated generic sentences from a raw text corpus. |
| Outcome: | The proposed approach reduces the top 1 error rate to 4% on negated LAMA dataset and improves on negating NLI benchmarks. |
Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing summarization systems can generate fluent summaries, but their ability to produce factually consistent summary remains questionable. |
| Approach: | They propose a framework that decomposes long texts into discourse-inspired chunks and utilizes discourse information to better aggregate sentence-level scores predicted by NLI models. |
| Outcome: | The proposed framework shows better performance over multiple benchmarks, focusing on long document summarization. |
Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021.naacl-main)
Copied to clipboard
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula
| Challenge: | Large-scale, open Natural Language Inference datasets have catalyzed the development of NLI models that exhibit close to human-level performance, but the use of these models for other downstream NLP tasks has met with limited success. |
| Approach: | They use multiple-choice reading comprehension and checking factual correctness of textual summarization tasks to investigate potential reasons for this . authors leverage abundance of data from reading comprehension datasets into longer-premise NLI datasets to improve their models . |
| Outcome: | The proposed models outperform models trained on converted datasets due to the difference in premise lengths. |
Fast and Accurate Factual Inconsistency Detection Over Long Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative AI models exhibit remarkable potential, however, hallucinations across various tasks present a significant challenge, particularly for longer inputs. |
| Approach: | They propose a task-agnostic model that uses large text chunks to condition over long texts and employ a novel algorithm to explain its decisions through relevant source sentence retrieval. |
| Outcome: | The proposed model outperforms existing methods on benchmarks and a new long-form dialogue dataset and surpasses competitive systems in efficiency and model explanation evaluations. |
Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge-to-text generators often struggle to faithfully generate descriptions for input facts . we propose a decoding-only method to reduce hallucinations . |
| Approach: | They propose a decoding-only method to generate accurate descriptions for input facts . they use a Natural Language Inference model as the model and replace it with a task-specific HVM . |
| Outcome: | The proposed method improves faithfulness with minimal impact on quality and in/out-of-distribution evaluations. |
Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing datasets focus on a single medium, information domain or specific application . authors propose novel methods for automated veracity assessment based on Natural Language Inference . |
| Approach: | They propose to build a PANACEA dataset that combines different data sources with different foci to ensure a unique set of claims. |
| Outcome: | The proposed methods are competitive with SOTA methods and provide a detailed discussion. |
Language Models for Lexical Inference in Context (2021.eacl-main)
Copied to clipboard
| Challenge: | Lexical inference in context (LIiC) is a variant of the natural language inference task focused on lexical semantics. |
| Approach: | They propose three approaches based on pretrained language models for LIiC . they propose a few-shot NLI classifier and a relation induction approach based upon handcrafted patterns expressing the semantics of lexical inference. |
| Outcome: | The proposed approaches outperform the previous state of the art and show their potential for LIiC. |
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained Masked Language Models are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. |
| Approach: | They propose a contrastive learning technique which turns pretrained MLMs into effective universal lexical and sentence encoders without additional data. |
| Outcome: | The proposed technique can turn MLMs into effective universal lexical and sentence encoders even without additional data. |
Embracing Ambiguity: Shifting the Training Target of NLI Models (2021.acl-short)
Copied to clipboard
| Challenge: | Previously, it was common to disregard ambiguity as noise or as a sign of poor quality data. |
| Approach: | They propose to train on the estimated label distribution of annotators in a NLI task . they use a learning loss based on this ambiguity distribution instead of gold-labels . |
| Outcome: | The proposed training method reduces divergence scores on a trial dataset . the study shows that targeting the ambiguity distribution can improve performance . |
EnerGIZAr: Leveraging GIZA++ for Effective Tokenizer Initialization (2025.findings-acl)
Copied to clipboard
| Challenge: | Continual pre-training has long been considered the default strategy for adapting models to non-English languages, but struggles with initializing new embeddings, especially for non-Latin scripts. |
| Approach: | They propose a method that leverages statistical word alignment techniques to improve continual pre-training by leveraging word alignment matrix between source and target tokens. |
| Outcome: | The proposed method outperforms existing methods on key NLP tasks including POS tagging, Sentiment Analysis, NLI, and NER in Hindi, Basque, Arabic and Korean. |
Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference (2022.emnlp-main)
Copied to clipboard
Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, Christopher Manning
| Challenge: | Existing frameworks for boosting consistency and accuracy of pre-trained NLP models without fine-tuning or re-training are lacking. |
| Approach: | They propose a framework for boosting the consistency and accuracy of pre-trained NLP models using pre-trainer natural language inference models without fine-tuning or re-training. |
| Outcome: | The proposed framework boosts consistency and accuracy of pre-trained NLP models using pre-train natural language inference models without fine-tuning or re-training. |
A synthetic data approach for domain generalization of NLI models (2024.acl-long)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset . |
| Approach: | They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings . |
| Outcome: | The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy. |
Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference (2022.findings-acl)
Copied to clipboard
| Challenge: | Previously, it's common to disregard it as noise or as a sign of poor-quality data, as their annotations are heavily based on personal experience and opinions. |
| Approach: | They propose to capture the human disagreement distribution from the perspective of model calibration. |
| Outcome: | The proposed model can achieve competitive performance when well-calibrated, on divergence scores between predictive probability and the true human opinion distribution, and the accuracy. |
VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work on label variation and annotation errors has focused on them in isolation. |
| Approach: | They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations. |
| Outcome: | The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items. |
Evaluating and Mitigating Inherent Linguistic Bias of African American English through Inference (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies show that NLP models trained on standard English produce biased outcomes against underrepresented English varieties. |
| Approach: | They propose a morphosyntactically-informed rule-based translation method that uses a greedy algorithm to debiase NLP models. |
| Outcome: | The proposed framework outperforms large language models while maintaining or improving the prediction performance. |
MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain (2021.acl-short)
Copied to clipboard
| Challenge: | a large number of crowdworker-constructed datasets have been used to conduct natural language inference (NLI) on unstructured, domainspecific texts such as patient notes, pathology reports, and scientific papers. |
| Approach: | They investigate whether MedNLI contains lexical and syntactic annotation artifacts associated with annotation process that allow hypothesis-only classifiers to achieve better-than-random performance. |
| Outcome: | The proposed model outperforms a majority-class baseline model on a physician-annotated dataset with premises extracted from clinical notes. |
NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals (2025.naacl-long)
Copied to clipboard
| Challenge: | Decomposing text into atomic propositions allows for finergrained inspection of text. |
| Approach: | They propose to decompose atomic propositions into atomic sub-problems that models must weigh when solving the overall problem. |
| Outcome: | The proposed method measures the inferential consistency of models and the diversity of examples in benchmark datasets. |
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances on neural approaches to natural language processing have triggered a resurgent interest on building intelligent open-domain chatbots. |
| Approach: | They propose a dialoguE COntradiction DEtection task and a conversational dataset . they show that their best contradiction detection model correlates well with human judgments . |
| Outcome: | The proposed model is more robust and generalizes well on analysis and out-of-distribution dialogues than standard (unstructured) Transformer models that explicitly hinge on utterance structures are more robust, the study shows . |
NormNet: Normalize Noun Phrases for More Robust NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to debunk false features in deep NLP models are inadequate . previous work suggests that models learn spurious features instead of the true signals of the task . |
| Approach: | They propose a normalization strategy to eliminate false features caused by the textual surfaces of noun phrases. |
| Outcome: | The proposed normalization strategy eliminates false features caused by the textual surfaces of noun phrases. |
Co-training for Low Resource Scientific Natural Language Inference (2024.acl-long)
Copied to clipboard
| Challenge: | Scientific Natural Language Inference (NLI) is a task to predict the semantic relation between sentences extracted from research articles. |
| Approach: | They propose a co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels. |
| Outcome: | The proposed method achieves an improvement of 1.5% in Macro F1 over the distant supervision baseline and substantial improvements over several other strong SSL baselines. |
Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | generating aspect-specific and general opinion summaries is challenging due to the lack of annotated data. |
| Approach: | They propose two unsupervised approaches to generate aspect-specific and general opinion summaries by training on synthetic datasets constructed with aspect-related review contents. |
| Outcome: | The proposed method outperforms existing methods on space and Oposum+ and on other metrics. |
Less for More: Enhanced Feedback-aligned Mixed LLMs for Molecule Caption Generation and Fine-Grained NLI Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent trends have led to the use of multimodal models to learn molecular and linguistic representations, either in separate but coordinated spaces or in a common space. |
| Approach: | They propose a novel atomic-level evaluation method leveraging off-the-shelf Natural Language Inference (NLI) models for use in the unseen chemical domain. |
| Outcome: | The proposed method surpasses state-of-the-art models in the unseen chemical domain while relying on a granularity-based evaluation method. |
To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. (2022.coling-1)
Copied to clipboard
| Challenge: | Reasoning and knowledge-related skills are considered as fundamental skills for natural language understanding (NLU) tasks. |
| Approach: | They propose a method to diagnose correlations between an NLU dataset and a specific skill. |
| Outcome: | The proposed method is able to diagnose correlations between dataset and logical reasoning skill on 8 MRC and 3 NLI datasets. |
Uncertainty-Aware Cross-Lingual Transfer with Pseudo Partial Labels (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained language models for zero-shot cross-lingual tasks are noisy and lack confidence. |
| Approach: | They propose an uncertainty-aware cross-lingual transfer framework with pseudo-partial-label to maximize the utilization of unlabeled data by reducing noise. |
| Outcome: | The proposed framework outperforms baselines on named entity recognition and natural language inference tasks on 40 languages. |
Combining Event Semantics and Degree Semantics for Natural Language Inference (2020.coling-main)
Copied to clipboard
| Challenge: | formal semantics has been used to account for the semantics of verb phrases and adverbial modifiers . but it is not clear whether these independent theories can be combined and extended to cases in which the phenomena in question interact. |
| Approach: | They propose a logic-based NLI system that combines event semantics and degree semantics. |
| Outcome: | The proposed system achieves high accuracies on linguistically challenging datasets . the proposed system can handle various combinations of linguistic phenomena without compromise . |
A Neural-Symbolic Approach to Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have enabled deep neural networks to perform natural language understanding tasks, but their performance can drastically deteriorate when logical reasoning is needed. |
| Approach: | They propose a framework for NLU based on analogical reasoning based upon neural processing and logical reasoning using both neural and symbolic processing. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two NLU tasks, question answering (QA) and natural language inference (NLI). |
A Deep Generative Approach to Native Language Identification (2020.coling-main)
Copied to clipboard
| Challenge: | Native language identification (NLI) is a multi-class classification task involving multiple features that capture the systematic fingerprints of the first language in the second language writing. |
| Approach: | They propose a deep generative language modelling approach to NLI that fine-tunes a GPT-2 model separately on texts written by the authors with the same L1 and assigns n-grams to an unseen text. |
| Outcome: | The proposed method outperforms traditional machine learning approaches and currently achieves the best results on the benchmark NLI datasets. |
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Contract review is a time-consuming procedure that costs companies millions of dollars each year . linguistic characteristics of contracts, such as negations by exceptions, contribute to the difficulty of this task . |
| Approach: | They propose a document-level natural language inference (NLI) task for contracts . they annotate and release the largest corpus to date consisting of 607 annotated contracts a linguistically rich system is proposed . |
| Outcome: | The proposed system is based on a contract review task that includes 607 annotated contracts. |
Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has explored the potential of leveraging natural language inference (NLI) techniques to enhance relation extraction (RE). |
| Approach: | They propose a method that verbalizes relation classes into class-indicative hypotheses to align a traditionally multi-class classification task to one of textual entailment. |
| Outcome: | The proposed method improves relation extraction performance on BioRED and ReTACRED. |
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. |
| Approach: | They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts . |
| Outcome: | The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts. |
For Generated Text, Is NLI-Neutral Text the Best Text? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a perfectly informative agent would eschew utterances that are redundant or contradict that which they have already said. |
| Approach: | They propose to use a pre-trained NLI model to assess whether a sentence entails, contradicts, or is neutral to prompt and preceding text. |
| Outcome: | The proposed model is predictive of generation errors made by GPT-3. |
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for assessing the validity of explanations for NLI are time-consuming and prone to logical errors. |
| Approach: | They propose a framework that integrates Large Language Models and Theorem Provers to verify and refine natural language explanations through crowd-sourcing . they propose to use TPs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI. |
| Outcome: | The proposed framework generates and formalises explanatory sentences and suggests potential inference strategies for NLI. |
Native Language Identification in Texts: A Survey (2024.naacl-long)
Copied to clipboard
| Challenge: | Native language identification is the task of automatically identifying an author’s native language (L1) based on their second language production. |
| Approach: | They present a survey of native language identification applied to texts . authors describe several text representations and computational techniques used in the task . |
| Outcome: | The proposed task has been widely studied for both text and speech, particularly for L2 English due to the availability of suitable corpora. |
Evaluating the Robustness of Discrete Prompts (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods that generate discrete prompts from a small set of training instances have reported superior performance, but manual writing prompts that generalize well is challenging due to several reasons. |
| Approach: | They propose to use discrete prompts to learn lexical constructs that would not be encountered in manually-written prompts. |
| Outcome: | The proposed method is robust against perturbations to NLI inputs but sensitive to other types of perturbations such as shuffling and deletion of prompt tokens. |
How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance. |
| Approach: | They propose a method for the automated creation of a challenging test set without relying on manual construction of artificial and unrealistic examples. |
| Outcome: | The proposed method reduces spurious correlations and improves model performance . examples labeled as having the highest difficulty show markedly decreased performance compared to the full dataset . |
Probing Linguistic Systematicity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evidence that deep natural language understanding models do not learn systematically is lacking. |
| Approach: | They examine whether deep natural language understanding models exhibit systematicity . they find that network architectures can generalize non-systematically . |
| Outcome: | The proposed model generalizes non-systematically, but is unsatisfactory, the authors argue . they show that the current state-of-the-art models do not generalize systematically . |
Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness (N18-1)
Copied to clipboard
| Challenge: | Currently, state-of-the-art models achieve impressive test set performance in the form of accuracy scores. |
| Approach: | They examine the models' robustness to semantically-valid alterations to the input data by identifying three factors and comparing their impact on three SNLI models. |
| Outcome: | The proposed models show that they can generalise to new in-domain instances while also showing that they suffer from insensitivity to small but semantically significant alterations. |
Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) |
| Approach: | They propose to use LLMs to probe their behavior using controlled experiments. |
| Outcome: | The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do. |
Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | Using a new architecture, alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Approach: | They propose a new architecture where alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning. |
| Outcome: | The proposed architecture achieves competitive performance on three popular benchmarks, SNLI, MultiNLI and SciTail, while maintaining lightweight parameter size. |
Convolutional Interaction Network for Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | Attention-based neural models have achieved great success in natural language inference (NLI). |
| Approach: | They propose a general model to capture the interaction between two sentences, which can be an alternative to the attention mechanism for NLI. |
| Outcome: | The proposed model can capture complex interactions on three large datasets. |
Lessons from Natural Language Inference in the Clinical Domain (D18-1)
Copied to clipboard
| Challenge: | State of the art models with deep neural networks lack generalization capabilities in specialized domains where training data is limited. |
| Approach: | They propose a dataset annotated by doctors performing a natural language inference task grounded in the medical history of patients. |
| Outcome: | The proposed model outperforms existing models in the clinical domain by incorporating domain knowledge from external data and lexical sources. |
KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches in NLP focus on “WHY A” rather than contrastive “WHA NOT B” Experimental results show that contrastive explanations are beneficial to fit the scenarios by clarifying the difference between the predicted answer and other possible wrong ones. |
| Approach: | They propose to generate contrastive explanations with counterfactual examples in NLI by identifying key phrases from input sentences and using them as key perturbations to generate counterfacts. |
| Outcome: | The proposed framework improves on SNLI and ETPA models by 91.9%. |
Conformal Predictor for Improving Zero-Shot Text Classification Efficiency (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been shown effective for zero-shot (0shot) text classification. |
| Approach: | They propose to limit the number of likely labels using a fast base classifier-based conformal predictor calibrated on samples labeled by the 0shot model. |
| Outcome: | The proposed models reduce the average inference time for NLI- and NSP-based models by 25.6% and 22.2% without dropping performance below the predefined error rate of 1%. |
Stress Test Evaluation for Natural Language Inference (C18-1)
Copied to clipboard
| Challenge: | Existing models perform well at standard datasets for NLI, achieving impressive results across different genres of text. |
| Approach: | They propose to use automatic stress tests to evaluate models' ability to make inferential decisions. |
| Outcome: | The proposed model performs well across genres of text, but lacks the ability to make inferential decisions. |
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)
Copied to clipboard
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith
| Challenge: | Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance. |
| Approach: | They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example. |
| Outcome: | The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. |
Syntactic Data Augmentation Increases Robustness to Inference Heuristics (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained neural models lack sensitivity to word order on controlled challenge sets . augmentation methods that improve accuracy on standard training sets may be a problem . |
| Approach: | They propose to augment standard training sets with syntactically informative examples by applying syntastic transformations to sentences from the MNLI corpus. |
| Outcome: | The proposed method improved BERT’s accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73 without affecting performance on the MNLI test set. |
BigNLI: Native Language Identification with Big Bird Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Native Language Identification (NLI) is a task that relies on time-consuming linguistic feature engineering and current transformer models are limited by input size. |
| Approach: | They propose to train a logistic regression classifier which only uses Big Bird embeddings to overcome this limitation. |
| Outcome: | The proposed method outperforms linguistic feature engineering models on the Reddit-L2 dataset and shows consistent out-of-sample and out-off-domain performance. |
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice. |
| Approach: | They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently. |
| Outcome: | The proposed models can detect errors and reduce them by reranking alternative summaries. |
Language Model Analysis for Ontology Subsumption Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on simple, triple-based, relational KBs but omit more sophisticated, logic-based conceptualised KB. |
| Approach: | They propose to use ontology subsumption axioms to probe LMs' knowledge of ontologies by probing datasets from atomic and complex concepts. |
| Outcome: | The proposed methods encode less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference but can improve on SI significantly when a small number of samples are given. |
Multilingual Summarization with Factual Consistency Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models generate factually inconsistent summaries, reducing their utility for real-world applications. |
| Approach: | They propose to use data filtering and controlled generation to detect hallucinations in machine generated summaries. |
| Outcome: | The proposed models detect factual inconsistencies in machine generated summaries, but they focus on English only. |
EENLP: Cross-lingual Eastern European NLP Index (2022.lrec-1)
Copied to clipboard
Alexey Tikhonov, Alex Malkhasov, Andrey Manoshin, George-Andrei Dima, Réka Cserháti, Md.Sadek Hossain Asif, Matt Sárdi
| Challenge: | Existing NLP resources for Eastern European languages are sparse. |
| Approach: | They propose to use existing Eastern European language resources to build cross-lingual datasets for five different semantic tasks to support commonsense reasoning. |
| Outcome: | The proposed model trains on 104 languages and shows impressive results on text analysis tasks. |
PECO: Examining Single Sentence Label Leakage in Natural Language Inference Datasets through Progressive Evaluation of Cluster Outliers (2023.eacl-main)
Copied to clipboard
| Challenge: | Efforts to debias NLI have led to datasets that exhibit different kinds of bias than those shown before. |
| Approach: | They propose a new technique to detect and reduce single sentence label leakage . leakage is a problem with many modern NLI datasets, they argue . future work must prioritize reducing this problem, they write . |
| Outcome: | a new model-driven technique can detect leakage and detect subpopulations in the datasets which exhibit it . the proposed technique is based on the progressive evaluation of cluster outliers (PECO) . it allows objective measurement of leakage, and automatic detection of subpopulations in the data which exhibit leakage. |
Neural Natural Language Inference Models Enhanced with External Knowledge (P18-1)
Copied to clipboard
| Challenge: | Existing datasets that allow for complex models to be trained are limited . if data is not available, can machines learn all knowledge needed to perform natural language inference? |
| Approach: | They propose to enrich neural natural language inference models with external knowledge . they propose to use this knowledge to build NLI models to leverage it . |
| Outcome: | The proposed models improve on the SNLI and MultiNLI datasets. |
How well do NLI models capture verb veridicality? (D19-1)
Copied to clipboard
| Challenge: | In natural language inference, contexts are considered veridical if they allow us to infer that their underlying propositions make true claims about the real world. |
| Approach: | They propose to use a dataset for veridicality evaluation consisting of 1,500 sentence pairs, covering 137 unique verbs. |
| Outcome: | The proposed model learns to make correct inferences about veridicality in verb-complement constructions. |
Deep Natural Language Feature Learning for Interpretable Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a small transformer language model, we can break down a complex task into a set of intermediary easier sub-tasks. |
| Approach: | They propose a method to break down a main task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
| Outcome: | The proposed method breaks down a complex task into a set of easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are weak and take advantage of failures and errors in datasets to improve performance. |
| Approach: | They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors. |
| Outcome: | The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks. |
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. |
| Approach: | They propose a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena using a set of datasets and an evaluation procedure for diagnosing how well a language model captures reasoning skills. |
| Outcome: | The proposed model can diagnose model behavior and verify model learning quality. |
XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)
Copied to clipboard
| Challenge: | XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages. |
| Approach: | They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches. |
| Outcome: | The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches. |
Contrastive Learning of Sentence Embeddings from Scratch (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings with unlabeled data are limited due to copyright restrictions, data distribution issues, and messy formats. |
| Approach: | They propose a contrastive learning framework that trains sentence embeddings with synthetic data. |
| Outcome: | The proposed framework produces positive and negative annotations given unlabeled sentences and generates sentences along with their corresponding annotations from scratch. |
Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI. |
| Approach: | They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases. |
| Outcome: | The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved . |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking (2025.findings-naacl)
Copied to clipboard
| Challenge: | FinNLI is a benchmark dataset for Financial Natural Language Inference (NLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. |
| Approach: | They propose to use FinNLI to evaluate financial natural language inference models across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. |
| Outcome: | The proposed dataset is based on a high-quality test set of 3,304 instances annotated by finance experts. |
Dually Self-Improved Counterfactual Data Augmentation Using Large Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generate counterfactual data augmentation are limited due to imbalance and biases in real-world training data. |
| Approach: | They propose a self-improved method for generating high-quality counterfacts using large language models. |
| Outcome: | The proposed method generates high-quality counterfacts on the natural language inference task using lightweight and task-specific LLMs. |
When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks. |
| Approach: | They conduct a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four different natural languages. |
| Outcome: | The proposed model exhibits decent cross-lingual zero-shot transfer, with no significant differences in word order and embedding alignment. |
Knowledge-Enhanced Evidence Retrieval for Counterargument Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models lack the reasoning abilities needed to find complex counterevidence. |
| Approach: | They propose a natural language inference model that finds counterevidence from diverse sources on the Web. |
| Outcome: | The proposed model outperforms baseline models for NLI tasks and finds complex counterevidence better. |
Asynchronous Deep Interaction Network for Natural Language Inference (D19-1)
Copied to clipboard
| Challenge: | Existing methods have framed the reasoning problem as a semantic matching task. |
| Approach: | They propose an asynchronous deep interaction network (ADIN) to deconstruct the reasoning process and implement asynchron and multi-step reasoning. |
| Outcome: | The proposed model outperforms strong baselines on three popular benchmarks: SNLI, MultiNLI, and SciTail. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
Towards Debiasing Translation Artifacts (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies show translation artifacts in translations influence performance of cross-lingual tasks. |
| Approach: | They propose a method to reduce translation artifacts by extending an established bias-removal technique. |
| Outcome: | The proposed method reduces translationese at sentence and word level . it is the first study to debias translations on a natural language inference task . |
Fact Verification on Knowledge Graph via Programmatic Graph Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fact verification on knowledge graphs use implicit reasoning to predict entailment between claims and KG triples. |
| Approach: | They propose a framework that integrates large language models for fact verification on knowledge graphs. |
| Outcome: | The proposed framework outperforms existing methods on knowledge graphs with 86.82% accuracy. |
DISCO: Distilling Counterfactuals with Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | high-quality counterfactual data is scarce for most tasks and not easily generated at scale. |
| Approach: | They propose a method for automatically generating high-quality counterfactual data at scale . they use a large general language model to generate phrasal perturbations and filter them . |
| Outcome: | The proposed method is task-agnostic and can be applied to the task of natural language inference. |
Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Ambiguity is embedded throughout natural language, and even simple utterances can have multiple interpretations when read in isolation. |
| Approach: | They propose a task-agnostic framework for evaluating a system’s ability to determine when to ask for clarification. |
| Outcome: | The proposed framework outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification. |
Plausible Extractive Rationalization through Semi-Supervised Entailment Signal (2024.findings-acl)
Copied to clipboard
| Challenge: | Abstract: Large language models are gaining widespread adoption in natural language processing tasks. |
| Approach: | They propose a semi-supervised approach to optimize for plausibility of extracted rationales by using a pre-trained natural language inference model and a supervised NLI predictor. |
| Outcome: | The proposed model outperforms unsupervised models by > 100% on a ERASER dataset. |
Mining Knowledge for Natural Language Inference from Wikipedia Categories (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Accurate lexical entailment (LE) and natural language inference (NLI) tasks require expensive annotations. |
| Approach: | They propose to pretrain Wikipedia categories for lexical entailment and natural language inference by pretraining them on WikiNLI and transferring them to other knowledge bases. |
| Outcome: | The proposed model can improve strong baselines such as BERT and RoBERTa by pretraining on WikiNLI and transferring the models on downstream tasks. |
OCNLI: Original Chinese Natural Language Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent efforts to extend natural language understanding to other languages have focused on (automatically) translating existing English datasets. |
| Approach: | They propose to use a Chinese dataset to generate annotated sentences from native speakers specializing in linguistics to elicit annotations. |
| Outcome: | The proposed dataset does not rely on automatic translation or non-expert annotation. instead, it elicits annotations from native speakers specializing in linguistics. |
SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinant from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. |
| Approach: | They propose a dataset that extracts social and behavioral determinants from clinical notes and uses them to form a natural language inference task. |
| Outcome: | The proposed dataset is based on publicly available notes and is more challenging than standard NLI benchmarks. |
Realistic Data Augmentation Framework for Enhancing Tabular Reasoning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to constructing training data for Natural Language Inference (NLI) tasks are expensive and time consuming. |
| Approach: | They propose a semi-automated framework for data augmentation for tabular inference . framework generates hypothesis templates transferable to similar tables . authors say framework could generate human-like tabular examples . |
| Outcome: | The proposed framework generates human-like tabular inference examples . it is based on human-written constraints and premise paraphrasing . |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
A Deep Transfer Learning Method for Cross-Lingual Natural Language Inference (2022.lrec-1)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) is a crucial task in AI and natural language processing. |
| Approach: | They propose an effective transfer learning approach for cross-lingual NLI . they perform experiments on English-Hindi language pairs in cross-linguistic setting . |
| Outcome: | The proposed model improves the baseline model by 10% over the state-of-the-art model. |
Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Multilingual transformers have been shown to have remarkable transfer skills in zero-shot settings. |
| Approach: | They investigate cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference using a large scale Chinese dataset. |
| Outcome: | The proposed model trains on Chinese and English natural language inference datasets. |
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | lexical overlap heuristics are effective for frequent example types but break down in more challenging cases. |
| Approach: | They propose to use a set of examples to test whether a sentence entails another . they propose to adopt three fallible syntactic heuristics for statistical NLI models . |
| Outcome: | The proposed model performs poorly on the HANS dataset, suggesting they have adopted these heuristics. |
Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Document-level Relation Extraction (DocRE) is a task that aims to extract relations from a long context. |
| Approach: | They propose an automated annotation method that integrates an LLM and a natural language inference module to generate relation triples. |
| Outcome: | The proposed method can extract relations from document-level relation datasets with minimal human effort. |
InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples (2022.coling-1)
Copied to clipboard
| Challenge: | InferES is an original corpus for Natural Language Inference (NLI) in European Spanish . |
| Approach: | They propose to implement and analyze a corpus-creating strategy utilizing expert linguists and crowd workers to provide high-quality data and facilitate the systematic evaluation of automated systems. |
| Outcome: | The proposed model obtains 72.8% accuracy and performs moderately well on negation-based adversarial examples. |
Partial-input baselines show that NLI models can ignore context, but they don’t. (2022.naacl-main)
Copied to clipboard
| Challenge: | Researchers have shown that many datasets contain statistical biases, or "annotation artifacts" that systems leverage to correctly predict entailment. |
| Approach: | They propose to use edited contexts to examine RoBERTa models' sensitivity to edited context to examine their model's sensitivity. |
| Outcome: | The proposed model can learn to condition on context, despite being trained on artifact-ridden datasets. |
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (N19-1)
Copied to clipboard
| Challenge: | Existing inference models that rely heavily on unsupervised single-word embeddings struggle to learn implied relationships between pairs of words. |
| Approach: | They propose to use word embeddings to learn and use background knowledge about implied relationships between words that are crucial for cross-sentence inference problems. |
| Outcome: | The proposed models gain 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI, and 8.8% on the adversarial SQu AD datasets. |
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual transformers are less effective in resource-lean scenarios and for distant languages. |
| Approach: | They propose to use massively multilingual transformers to pretrain languages . they show that MMTs are less effective in resource-lean scenarios and distant languages if they are pre-trained via language modeling . |
| Outcome: | The proposed model is less effective in resource-lean scenarios and for distant languages than cross-lingual word embeddings. |
Dialogue Natural Language Inference (P19-1)
Copied to clipboard
| Challenge: | Consistency is a long standing issue faced by dialogue models. |
| Approach: | They propose to frame the consistency of dialogue agents as natural language inference and create a new natural language dataset called Dialogue NLI. |
| Outcome: | The proposed model can improve the consistency of a dialogue model with human evaluation and automatic metrics on a suite of evaluation sets designed to measure the model’s consistency. |
Temporal Reasoning in Natural Language Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | We use five new natural language inference (NLI) datasets focused on temporal reasoning. |
| Approach: | They introduce five new natural language inference datasets focused on temporal reasoning. |
| Outcome: | The proposed models capture the temporal reasoning of four existing datasets. |
Recursive Routing Networks: Learning to Compose Modules for Language Understanding (N19-1)
Copied to clipboard
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, Lauri Karttunen
| Challenge: | Recursive Routing Networks are modular, adaptable models that learn effectively in diverse environments. |
| Approach: | They propose to apply Recursive Routing Networks (RRNs) to natural language understanding by integrating them into existing architectures and recurrent network hidden layers. |
| Outcome: | The proposed model optimizes the parameters of the functions and the meta-learner decision-making component for routing inputs through those functions. |
LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new benchmark for natural language inference is proposed for long premises . lawngNLI can train and test systems for implication-based case retrieval and argumentation. |
| Approach: | They propose a new natural language inference benchmark LawngNLI from U.S. legal opinions with automatic labels with high human-validated accuracy. |
| Outcome: | The proposed benchmark can train and test systems for implication-based case retrieval and argumentation. |
Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity (2024.lrec-main)
Copied to clipboard
| Challenge: | Using labeled NLI datasets for learning sentence embeddings leads to improved performance for natural language understanding tasks. |
| Approach: | They compare two data augmentation techniques for learning better sentence embeddings . they use a cross-lingual transfer technique that exploits English resources as training data to yield non-English sentence embeds as zero-shot inference . |
| Outcome: | The proposed techniques yield better performance on Japanese and Korean sentences. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using a poison signature, attackers can manipulate training data to manipulate the target class at test time. |
| Approach: | They propose a backdoor poisoning attack that generates poisoned training samples by poison injection in latent space and a conditional adversarially regularized autoencoder to generate poisones. |
| Outcome: | The proposed attack generates poisoned training samples by poison injection in latent space and shows that the target class can be steered to the poison class with success rates of >80% when the input hypothesis is injected with the poison signature. |
BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Biomedical research has progressed at a tremendous pace, with PubMed2 indexing well over 1M publications per year in the past eight years. |
| Approach: | They propose a semi-supervised procedure that bootstraps biomedical NLI datasets from positive entailment examples present in biomedically published texts. |
| Outcome: | The proposed procedure bootstraps biomedical NLI datasets from positive entailment examples from biomedically challenging texts. |
Meta-training with Demonstration Retrieval for Efficient Few-shot Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models have impressive fewshot performance on many NLP tasks and domains. |
| Approach: | They propose a meta-training approach that uses demonstration retrieval to train parameter-efficient models that generalize well on a larger variety of tasks. |
| Outcome: | The proposed approach outperforms many parameter-efficient methods on QA, NLI, and text classification tasks. |
Contract Discovery: Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines (2020.findings-emnlp)
Copied to clipboard
Łukasz Borchmann, Dawid Wisniewski, Andrzej Gretkowski, Izabela Kosmala, Dawid Jurkiewicz, Łukasz Szałkiewicz, Gabriela Pałka, Karol Kaczmarek, Agnieszka Kaliska, Filip Graliński
| Challenge: | Existing methods for detecting text fragments are not suitable for contract discovery, since it requires manual definition of a few examples, followed by conventional information. |
| Approach: | They propose a task where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. |
| Outcome: | The proposed task differs substantially from conventional NLI and shared tasks on legal information extraction. |
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. |
| Approach: | They propose an infusion approach that partitions a large knowledge graph into smaller sub-graphs and infuses their specific knowledge into various BERT models using lightweight adapters. |
| Outcome: | The proposed approach improves the underlying BERTs and achieves new SOTA performance on six downstream tasks. |
MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models fail to capture important semantic features of logic such as monotonicity and negation. |
| Approach: | They propose a modular step-by-step approach to natural language inference . they use a language model to generate edits to incrementally transform the premise into the hypothesis . |
| Outcome: | The proposed method outperforms baseline models in realistic cross-domain settings with improvements up to 12.6% (relative). |
The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution (P19-1)
Copied to clipboard
| Challenge: | Existing methods for coreference resolution exploit the number and gender of antecedents or have been handcrafted and do not reflect the diversity of naturally occurring text. |
| Approach: | They propose a trick to improve resolution by antecedent switching to target common-sense understanding and world knowledge. |
| Outcome: | The proposed method achieves state-of-the-art results on the GAP coreference task. |
Identifying inherent disagreement in natural language inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Natural language inference is the task of determining whether text is entailed, contradicted or unrelated to another piece of text. |
| Approach: | They propose to tease systematic inferences from disagreement items by capturing modes in annotations to simulate uncertainty in the annotation process. |
| Outcome: | The proposed approach performs statistically better than baselines on the CommitmentBank corpus in English. |
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing BERT-based pre-trained language models achieve high performance on many downstream tasks, but native derived sentence representations are collapsed and thus poor performance on semantic textual similarity (STS) tasks. |
| Approach: | They propose a framework for self-supervised Sentence Representation Transfer that adopts contrastive learning to fine-tune BERT in an unsupervised way. |
| Outcome: | The proposed framework improves on the BERT-derived representations by 8% on STS datasets and shows robustness in data scarcity scenarios. |
NonFactS: NonFactual Summary Generation for Factuality Evaluation in Document Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained abstractive summarization models generate fluent summaries that are inconsistent with context document and contain nonfactual information. |
| Approach: | They propose a data generation model that synthesizes nonfactual summaries using human annotations. |
| Outcome: | The proposed model can generate nonfactual summaries and generalize to out-of-domain documents. |
Efficient PRM Training Data Synthesis via Formal Verification (2026.findings-acl)
Copied to clipboard
Ryo Kamoi, Yusen Zhang, Nan Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Wenpeng Yin, Rui Zhang
| Challenge: | Existing approaches for constructing PRM training data rely on human annotation or sampling-based labeling methods that require repeated LLM calls. |
| Approach: | They propose a framework that synthesizes PRM training data by annotating step-level error labels using formal verification tools such as Z3 and Isabelle. |
| Outcome: | The proposed framework synthesizes PRM training data from formal logic and theorem proving tasks without human annotation or additional LLM calls. |
Using Natural Language Relations between Answer Choices for Machine Comprehension (N19-1)
Copied to clipboard
| Challenge: | Current approaches to the reading comprehension task quantify the relationship between each question and answer choice independently and pick the highest scoring option. |
| Approach: | They propose a method to leverage natural language relations between answer choices to improve machine comprehension. |
| Outcome: | The proposed model improves the performance of a reading comprehension task by leveraging natural language relations between answer choices. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020.emnlp-main)
Copied to clipboard
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, Caiming Xiong
| Challenge: | Existing work on few-shot intent classification without OOS has focused on the few-shot intent classification with out-of-scope intents. |
| Approach: | They propose to use BERT-style pairwise encoding to train a binary classifier that estimates the best matched training example for a user input. |
| Outcome: | The proposed approach achieves more stable and accurate in-domain and OOS detection accuracy than RoBERTa-based classifiers and embedding-based nearest neighbor approaches. |
LCHAIM - Investigating Long Context Reasoning in Hebrew (2025.findings-acl)
Copied to clipboard
| Challenge: | a dataset designed to evaluate long premises and complex reasoning models in Hebrew is being used in research. |
| Approach: | They introduce a dataset to evaluate Hebrew NLI models on long premises and complex reasoning tasks. |
| Outcome: | The proposed dataset evaluates models on tasks involving long premises and complex reasoning. |
Alignment Rationale for Natural Language Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Existing explanation methods pick prominent features, but alignments between words or phrases are more enlightening clues to explain the model. |
| Approach: | They propose a method to generate alignment rationale explanations for co-attention based models in NLI by feature selection. |
| Outcome: | The proposed method is more faithful and human-readable compared with existing methods. |
Thinking Like a Skeptic: Defeasible Inference in Natural Language (2020.findings-emnlp)
Copied to clipboard
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, Yejin Choi
| Challenge: | Defeasible inference is a mode of reasoning in which an inference may be weakened or overturned in light of new evidence. |
| Approach: | They propose a dataset for defeasible inference in natural language that includes extensions to existing inference datasets. |
| Outcome: | Defeasible NLI extends existing datasets for defeaasibility inference in natural language . generative models can weaken or strengthen inferences up to 68% of the time, it shows . |
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning. |
| Approach: | They propose to use a common framework to solve commonsense reasoning tasks using a dataset from NLI. |
| Outcome: | The proposed method achieves state-of-the-art unsupervised performance on two commonsense reasoning tasks. |
Robust Native Language Identification through Agentic Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often achieve high performance by leveraging superficial contextual clues rather than the underlying linguistic patterns indicative of native language (L1) influence. |
| Approach: | They propose an agentic NLI pipeline where specialized agents accumulate and categorize diverse linguistic evidence before an independent final assessment. |
| Outcome: | The proposed pipeline significantly improves robustness against misleading contextual clues and performance consistency compared to standard prompting methods. |
On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Hallucination is a popular topic in natural language generation (NLG). |
| Approach: | They propose to use large language models to evaluate faithfulness of guided NLGs by a rubric template and large language inference models to score the generation on quantifiable scales. |
| Outcome: | The proposed system can provide accurate judgement and explain whether a source and generation are factually consistent. |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference (2022.naacl-main)
Copied to clipboard
| Challenge: | a method for learning an NLI model is time-consuming and resource-intensive, but it can save time and resources. |
| Approach: | They propose a method for predicting model performance without fine-tuning it . they compare sentence embeddings with cosine similarity to classifiers . |
| Outcome: | The proposed method can save time and resources by comparing pre-trained models to real-world datasets. |
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020.findings-emnlp)
Copied to clipboard
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | In this paper, we present NLP resources for 11 major Indian languages . distributional representations are the cornerstone of modern NLP, authors say . |
| Approach: | They introduce NLP resources for 11 major Indian languages from two major language families . monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . they also compile a benchmark for Indian language NLU to evaluate their results . |
| Outcome: | The monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . the pre-trained language models are based on the compact ALBERT model . |
Validity Assessment of Legal Will Statements as Natural Language Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | This study introduces a dataset that focuses on the validity of statements in legal wills. |
| Approach: | They propose a dataset that focuses on the validity of statements in legal wills. |
| Outcome: | The proposed model achieves 80% macro F1 and accuracy, but group accuracy is in mid 80s at best, suggesting that the models’ understanding of the task remains superficial. |
Enhancing Generalization in Natural Language Inference by Syntax (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have the state-of-the-art performance on natural language inference (NLI). |
| Approach: | They propose to use dependency trees to enhance generalization of BERT in a natural language inference task by leveraging on a graph convolutional network to represent a syntax-based matching graph with heterogeneous matching patterns. |
| Outcome: | The proposed method makes BERT more robust on syntactic changes. |
On Reference (In-)Determinacy in Natural Language Inference (2025.findings-naacl)
Copied to clipboard
Sihao Chen, Chaitanya Malaviya, Alex Fabrikant, Hagai Taitelbaum, Tal Schuster, Senaka Buthpitiya, Dan Roth
| Challenge: | Using reference determinacy, models fail to recognize context mismatch in NLI examples . |
| Approach: | They propose a benchmark to identify reference ambiguity in NLI examples . they propose RD as a possible assumption for natural language inference . |
| Outcome: | The proposed benchmark identifies reference ambiguity in natural language inference examples . 80% false contradiction and >50% entailment predictions are found . |
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt Tuning has been successful as a parameter-efficient method of conditioning large-scale pre-trained language models to perform downstream tasks. |
| Approach: | They propose to use a vector-quantized input-contextualized prompt as an extension to the soft prompt tuning framework to learn contextualization of soft prompt tokens. |
| Outcome: | The proposed prompt outperforms soft prompt tuning by an average margin of 1.19% on various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI. |
Continual Few-Shot Learning for Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large number of end-to-end systems are needed for many tasks in natural language processing. |
| Approach: | They propose a continual few-shot learning task where a system is asked to correct mistakes with a few training examples. |
| Outcome: | The proposed task compares two NLI and one sentiment analysis datasets with baselines from diverse paradigms. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |
Evaluating Numeracy of Language Models as a Natural Language Inference Task (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enhanced their capabilities to solve mathematical problems, but other aspects of numeracy remain underexplored. |
| Approach: | They propose to frame numeracy as a Natural Language Inference task to assess the models’ ability to understand both numbers and language contexts. |
| Outcome: | The proposed model outperforms smaller models in arithmetic tasks, indicating that mathematical reasoning cannot be generalized to other numeracy skills such as number comparison and normalization. |
Pairwise Supervised Contrastive Learning of Sentence Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to improve sentence representation learning have a common weakness . siamese or triplet loss only learns from individual sentence pairs or tripletes . |
| Approach: | They propose a discrimination-based approach to bridge entailment and contradiction understanding with categorical concept encoding. |
| Outcome: | The proposed method outperforms the state-of-the-art method on downstream tasks . it improves 10%–13% on clustering tasks and 5%–6% on STS tasks compared with the previous method . |
Learning to Rank for Plausible Plausibility (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to encoding sentences using contextualized encoders are inconsistent . |
| Approach: | They propose to use a cross entropy log-loss objective to improve plausibility . they propose a margin-based loss leads to a more plausible model of plausability . |
| Outcome: | The proposed loss is intuitively wrong when applied to plausibility tasks . the proposed loss leads to a more plausible model of plausability . |
Semantic-Eval : A Semantic Comprehension Evaluation Framework for Large Language Models Generation without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have emerged as key drivers of progress in the field of natural language processing. |
| Approach: | They propose a framework that assesses LLM-generated text based on semantic understanding. |
| Outcome: | The proposed framework surpasses traditional evaluation metrics and lags behind GPT-4. |
Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning (2022.coling-1)
Copied to clipboard
Xuantao Lu, Jingping Liu, Zhouhong Gu, Hanwen Tong, Chenhao Xie, Junyang Huang, Yanghua Xiao, Wenguang Wang
| Challenge: | Existing methods to parse natural language into structured logical expressions have limitations due to paucity of labeled data. |
| Approach: | They propose a scoring model to automatically learn a model-based reward . they also propose introducing a Chinese-PL/FOL dataset to compensate for paucity of labeled data . |
| Outcome: | The proposed model outperforms competitors on several datasets. |
FLUTE: Figurative Language Understanding through Textual Explanations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Figurative language understanding is a recognizing textual entailment task, but lacks data for figurative language. |
| Approach: | They propose to use a dataset to analyze figurative NLI instances with explanations to improve models' performance. |
| Outcome: | The proposed dataset can scale up models even for figurative language using human annotations. |
ConGen: Unsupervised Control and Generalization Distillation For Sentence Representation (2022.findings-emnlp)
Copied to clipboard
Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Lalita Lowphansirikul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Sentence representations are essential in many NLP tasks operating at the sentence level. |
| Approach: | They propose an unsupervised sentence representation method to reduce the supervised-unsupervised performance gap for smaller models. |
| Outcome: | The proposed method outperforms supervised training on STS, text classification, and natural language inference tasks on smaller models. |
MPII: Multi-Level Mutual Promotion for Inference and Interpretation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for providing interpretations provide human-unfriendly interpretations, resulting in sub-optimal performance. |
| Approach: | They propose a multi-level Mutual Promotion mechanism for self-evolved inference and sentence-level interpretation that integrates inference with interpretation in an autoregressive manner. |
| Outcome: | The proposed approach outperforms baseline models on NLI and CQA tasks for both inference performance and interpretation quality. |
What does the Failure to Reason with “Respectively” in Zero/Few-Shot Settings Tell Us about Language Models? (2023.acl-long)
Copied to clipboard
| Challenge: | In the context of natural language inference, we examine how language models reason with respective readings from two perspectives: syntactic-semantic and commonsense-world knowledge. |
| Approach: | They propose a controlled synthetic dataset WikiResNLI and a naturally occurring dataset NatResLI to encompass various explicit and implicit realizations of "respectively". |
| Outcome: | The proposed datasets include explicit and implicit readings of "respectively" the proposed dataset shows that fine-tuned models struggle with understanding readings without explicit supervision. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study (D18-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate deep learning models that are not considered for test set accuracy are difficult to interpret. |
| Approach: | They examine the impact of a test set question’s difficulty to determine if there is a relationship between difficulty and performance. |
| Outcome: | The proposed model can learn examples of varying difficulty at different rates if it does well on hard examples and poor on easy items because a dataset is all easy, but has "solved" anything? |
Universal Sentence Representation Learning with Conditional Masked Language Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn sentence representations on unlabeled corpora are difficult and expensive to obtain, making it hard to cover many domains and languages. |
| Approach: | They propose a method to train sentence representations on large unlabeled corpora by conditioning on the encoded vectors of adjacent sentences. |
| Outcome: | The proposed method outperforms existing models on SentEval and can be extended to a broad range of languages and domains. |
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets are often flooded with repetitive and spurious patterns, leading to a lack of linguistic diversity. |
| Approach: | They propose a method that uses cartography to automatically identify and filter examples that demonstrate challenging reasoning patterns and then automatically compose new ones with similar patterns. |
| Outcome: | The proposed approach improves performance on eight out-of-domain test sets compared to training on the 4x larger MultiNLI dataset. |
Culturally Aware Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Cultural norms are behavioral rules and conventions shared within specific groups, connecting cultural symbols and values. |
| Approach: | They propose a task that operationalizes cultural variations in language understanding through a natural language inference task that surfaces cultural variations as label disagreement between annotators from different cultural groups. |
| Outcome: | The proposed model can be evaluated at which levels it is culturally aware. |
SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Natural Language Inference (NLI) datasets are not related to scientific text. |
| Approach: | They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. |
| Outcome: | The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%. |
Towards Distribution-shift Robust Text Classification of Emotional Content (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that supervised models are more robust to change in domain and distribution, but the decrease in performance due to the distribution shift is still a major issue for supervised systems. |
| Approach: | They propose to fine-tune supervised models on task-specific datasets to achieve out-of-distribution performance. |
| Outcome: | The proposed model outperforms all available models in distribution and out of distribution with only a few thousand training samples. |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
Finding a Balanced Degree of Automation for Summary Evaluation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Automated summarization metrics are reliable but often poorly correlated with human judgment. |
| Approach: | They propose a semi-automatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. |
| Outcome: | The proposed metrics are semi-automatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. |
When data permutations are pathological: the case of neural natural language inference (D18-1)
Copied to clipboard
| Challenge: | Recent research indicates that these datasets are trivially annotated to a large degree. |
| Approach: | They propose to use a trending NLP task to test a neural architecture for NLI . they show a pathological sensitivity of the systems to permutations of the training set . |
| Outcome: | The proposed model outperforms the baseline model by multiple percentage points . the proposed model is based on the proposed dataset and the soundness of the task . |
Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pre-trained language models have achieved remarkable zero-shot performance . we propose a self-learning framework that utilizes unlabeled data of target languages . |
| Approach: | They propose a self-learning framework that utilizes unlabeled data of target languages to select silver labels for cross-lingual transfer tasks. |
| Outcome: | The proposed framework outperforms baseline models on two cross-lingual tasks by 10 F1 on average and 2.5 accuracy on natural language inference (NLI). |
Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Sentence embedding is essential for many NLP tasks, but reliance on manual labels limits scalability. |
| Approach: | They propose a method for controlling the generation direction of large language models in the latent space by integrating ranking information and semantic information. |
| Outcome: | The proposed method achieves new SOTA performance with a modest cost in ranking sentence synthesis. |
NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic (2023.acl-long)
Copied to clipboard
| Challenge: | Despite the recent advances in distributed representation and neural networks, it remains an open question whether the models perform real reasoning to reach their conclusions or rely on spurious correlations. |
| Approach: | They propose to use logic formalism to perform systematic attacks centring around natural logic to generate better adversarial examples with fewer visits to the victim models. |
| Outcome: | The proposed framework generates better adversarial examples with fewer visits to the victim models. |
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) Large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs, but collecting explanations for every label is still time-consuming. |
| Approach: | They propose to use Large Language Models (LLMs) as annotators to generate model explanations for a few given human labels. |
| Outcome: | The proposed models can generate human-provided explanations from human labels, but they are still time-consuming. |
PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence . |
| Approach: | They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually. |
| Outcome: | The proposed dataset can be used to understand the compositionality of NLI labels. |
Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate gender bias in PLMs focus on one label out of three labels, such as neutral. |
| Approach: | They propose a bias evaluation method for PLMs that considers all the three labels of NLI task and then defines a measure based on the corresponding label output. |
| Outcome: | The proposed method can distinguish biased, incorrect inferences from non-biased incorrect infertility better than baseline, resulting in a more accurate bias evaluation. |
Evaluating Generative Language Models in Information Extraction as Subjective Question Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Modern large language models (LLMs) perform poorly in elementary tasks like relation extraction and event extraction due to two issues in conventional evaluation methods. |
| Approach: | They propose a method to evaluate large language models by incorporating a human annotation schema. |
| Outcome: | The proposed evaluation method improves matching between model outputs and golden labels. |
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction (2023.acl-long)
Copied to clipboard
Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster
| Challenge: | In many NLP tasks, the input text can be seen as a sequence of related segments. |
| Approach: | They propose a layer-adjustable interactions framework that contextualizes token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. |
| Outcome: | The proposed model reduces 30-50% of attention FLOPs while maintaining high accuracy. |
Knowledge-Enhanced Natural Language Inference Based on Knowledge Graphs (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to natural language inference rely on semantic knowledge, but background knowledge is limited to a few specific types. |
| Approach: | They propose a Knowledge Graph-enhanced NLI model that leverages background knowledge stored in knowledge graphs to facilitate inference. |
| Outcome: | The proposed model can leverage background knowledge stored in knowledge graphs to perform the task. |
Tokenization is Sensitive to Language Variation (2025.findings-acl)
Copied to clipboard
| Challenge: | Variation in language is often linked to regional, social, and contextual factors. |
| Approach: | They propose a method to estimate tokenizer impact on downstream LLM performance . they pre-train BERT models with the popular Byte-Pair Encoding algorithm . |
| Outcome: | The proposed model improves on Rényi efficiency and other metrics on language variation. |
Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evidence that humans make numerous inferences to understand discourse and text is not fully understood. |
| Approach: | They propose to use textual inference datasets with multi-sentence premises to solve the entailment verification problem. |
| Outcome: | The proposed model outperforms GPT-3.5 and rivals GPL-4 in EV tasks. |
Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for factual consistency in knowledge-grounded dialogues are unreliable and limit their applicability. |
| Approach: | They propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue using automatic question generation and question answering. |
| Outcome: | The proposed evaluation metric consistently shows higher correlation with human judgements. |
Evaluating BERT for natural language inference: A case study on the CommitmentBank (D19-1)
Copied to clipboard
| Challenge: | Natural language inference datasets can identify premise-hypothesis relationship without observing premise . recasting of the CommitmentBank for NLI creates hypotheses that stand in entailment/contradiction/neutral relationship with premise. |
| Approach: | They propose to recast the CommitmentBank for NLI to stand in certain relationships with the premise . hypotheses are complements of clause-embedding verbs in each premise, rethinking the CommittedBank . |
| Outcome: | The proposed model performs well on the CommitmentBank with 85% F1 . however, the model does not capture the full complexity of pragmatic reasoning, authors say . |
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate factual consistency of text depend on limited data . e.g., generated text can contain factual inconsistencies that are irrelevant to context . |
| Approach: | They propose a new holistic metric that measures factual inconsistencies . they use 4.7M training examples from 7 well-established tasks . |
| Outcome: | The proposed metric outperforms existing metrics on 22 datasets and matches or outperFORMs them. |
Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity. |
| Approach: | They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods. |
| Outcome: | The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset. |
Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown advantages of generative classifiers in terms of data efficiency and robustness. |
| Approach: | They propose a generative classifier for natural language inference (NLI) they compare it to discriminative models and large-scale pretrained models like BERT . |
| Outcome: | The proposed classifier outperforms discriminative and pretrained baselines across several challenging NLI experimental settings, including small training sets, imbalanced label distributions, and label noise. |
New Protocols and Negative Results for Textual Entailment Data Collection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language inference data has proven useful in benchmarking and as pretraining data for tasks requiring language understanding. |
| Approach: | They propose four alternative protocols to improve annotation quality and diversity . they use 8.5k-example training sets to compare different protocols . |
| Outcome: | The proposed protocols improve the ease of training and quality of the examples. |
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural network models have significantly pushed forward performance on natural language processing benchmarks with the development of largescale language model pre-training. |
| Approach: | They find that models on natural language inference and reading comprehension are unstable . they propose to use a model-selection routine to analyze the model's instability . |
| Outcome: | The proposed models can perform poorly on two language-related tasks, the authors show . they also show that the model selection routine is unstable, and that it is not reliable . |
Data and Representation for Turkish Natural Language Inference (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large annotated datasets in NLP are overwhelmingly in English . obtaining new annotation resources for each task in each language would be prohibitively expensive . |
| Approach: | They propose to use machine translation to translate large annotated datasets into Turkish . they find that in-language embeddings are essential and morphological parsing can be avoided . |
| Outcome: | The proposed model trains on human-translated evaluation sets. |
MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining (2026.findings-acl)
Copied to clipboard
Phung Gia Huy, Hai An Vu, Minh-Phuc Truong, Thang Duc Tran, Linh Ngo Van, Thanh Hong Nguyen, Trung Le
| Challenge: | Existing approaches to train dense representations require explicit coordination of how information is arranged across embedding dimensionality and model depth. |
| Approach: | They propose a framework that trains Matryoshka representations using self-distilled intra-relational alignment and Progressive information chaining. |
| Outcome: | The proposed framework produces coherent and compact Matryoshka representations with significant performance advantages under low-dimensional models. |
GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection? (2024.lrec-main)
Copied to clipboard
| Challenge: | HateCheck test cases are generic and have simplistic sentence structures that do not match the real-world data. |
| Approach: | They propose a framework to generate more diverse and realistic functional tests from scratch by instructing large language models. |
| Outcome: | The proposed framework generates more diverse and realistic functional tests from scratch by instructing large language models (LLMs). |
Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study of presupposition, discourse and sarcasm suggests that pre-finetuning can improve models' performance on presimplified cases. |
| Approach: | They propose to leverage the connection between presupposition, discourse and sarcasm to improve models' performance. |
| Outcome: | The proposed model improves on cases involving presupposition by pre-finetuning on additional tasks and datasets. |
Neutralizing Bias in LLM Reasoning using Entailment Graphs (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) is a foundational understanding task in language understanding. |
| Approach: | They propose a framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias. |
| Outcome: | The proposed framework reduces hallucinations from attestation bias on original and bias-neutralized datasets while keeping hypotheses unchanged. |
Ecologically Valid Explanations for Label Variation in NLI (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation exists in many natural language processing tasks, including NLI . |
| Approach: | They build an English dataset of 1,415 ecologically valid explanations for 122 MNLI items . they find that people can systematically vary on their interpretation . |
| Outcome: | The proposed dataset contains 1,415 ecologically valid explanations for 122 items . the results show that people can vary on interpretation and highlight differences . |
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are ineffective in addressing the reverse word-overlap bias. |
| Approach: | They propose to investigate the reverse word-overlap bias in NLI models . they find that existing debiasing methods are generally ineffective . |
| Outcome: | The proposed model is biased towards the non-entailment label on instances with low overlap . the proposed model does not have minority examples, the authors show . |
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work highlights the fallacies of zero-shot cross-lingual transfer with large multilingual models. |
| Approach: | They propose to replace sequential fine-tuning with joint fine-uning on source and target language instances. |
| Outcome: | The proposed techniques yield improved and more stable FS-XLT across the board. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
IndicXNLI: Evaluating Multilingual Inference for Indian Languages (2022.emnlp-main)
Copied to clipboard
| Challenge: | Indic NLP has made rapid advances in terms of corpora and pre-trained models, but benchmark datasets on standard NLU tasks are limited. |
| Approach: | They propose to use an NLI dataset for 11 Indic languages to test their accuracy. |
| Outcome: | The proposed dataset provides useful insights into the behaviour of pre-trained models for a diverse set of languages. |
Tracing L1 Interference in English Learner Writing: A Longitudinal Corpus with Error Annotations (2025.emnlp-main)
Copied to clipboard
Poorvi Acharya, J. Elizabeth Liebl, Dhiman Goswami, Kai North, Marcos Zampieri, Antonios Anastasopoulos
| Challenge: | high-quality learner corpora are rarely available for studies of second language acquisition and language transfer. |
| Approach: | They propose to curate a corpus of adult learners with longitudinal data that includes 15 different L1s. |
| Outcome: | The proposed corpus contains 687 texts written by adult learners in the USA . authors show that the corpus can be used to explore language learning trajectories over time. |
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is an increasingly important task for natural language understanding . however, the ability of NLI models to make pragmatic inferences remains understudied . |
| Approach: | They use semi-automatically generated sentence pairs to evaluate whether NLI models make pragmatic inferences. |
| Outcome: | The proposed model trains on multiNLI and shows that it learns to draw pragmatic inferences. |
NILE : Natural Language Inference with Faithful Natural Language Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | Recent growth in popularity of deep learning models on NLP classification tasks has accompanied the need for generating some form of natural language explanation of predicted labels. |
| Approach: | They propose a novel method which generates labels along with its faithful explanations. |
| Outcome: | The proposed method is more accurate than previously reported methods and has higher sensitivity than previous methods. |
Towards Robustifying NLI Models Against Lexical Dataset Biases (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show that deep learning models exploit dataset biases without deep understanding of the language semantics. |
| Approach: | They propose two methods to debiase models against lexical dataset biases . they use contradiction-word bias and word-overlapping bias as examples . |
| Outcome: | The proposed method removes label bias at embedding level, while the other uses a bag-of-words sub-model to capture features likely to exploit the bias. |
Uncertain Natural Language Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Existing categorical bin assignments for NLI are not effective for entailment inference. |
| Approach: | They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments. |
| Outcome: | The proposed model can be used in pre-training and correlates well with humans. |
Mitigating Spurious Correlation in Natural Language Understanding with Counterfactual Inference (2022.emnlp-main)
Copied to clipboard
Can Udomcharoenchaikit, Wuttikorn Ponwitayarat, Patomporn Payoungkhamdee, Kanruethai Masuk, Weerayut Buaphet, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Existing approaches to debias NLU models rely on superficial patterns to produce correct predictions . lexical overlap and annotation artifacts can be used to make shortcuts . |
| Approach: | They propose a causal analysis framework to help debias NLU models by defining causal relationships and utilizing counterfactual inference to mitigate bias. |
| Outcome: | The proposed framework can improve robustness across three NLU tasks while maintaining high in-distribution performance. |
Explaining Interactions Between Text Spans (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing highlight-based explanations focus on identifying individual important features or interactions only between adjacent tokens or tuples of tokens. |
| Approach: | They propose a multi-annotator dataset of human span interaction explanations for NLU and FC. |
| Outcome: | The proposed method compares human reasoning processes to those of a fine-tuned large language model. |
InferBR: A Natural Language Inference Dataset in Portuguese (2024.lrec-main)
Copied to clipboard
| Challenge: | Portuguese has few NLI-annotated datasets created through automatic translation followed by manual checking. |
| Approach: | They propose to generate premises and hypotheses using a semiautomatic process to generate sentences and manually check the annotations. |
| Outcome: | The proposed dataset is better at recognizing entailment classes in other Portuguese datasets than the reverse. |
Improving the robustness of NLI models with minimax training (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our method consistently outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets, while maintaining high in-distance accuracy. |
| Approach: | They propose a minimax objective between a learner model being trained for the task and an auxiliary model aiming to maximize the learner's loss by up-weighting underrepresented "hard" examples with patterns that contradict the shortcuts learned from the prevailing "easy" examples. |
| Outcome: | The proposed method outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets while maintaining high in-distance accuracy. |
Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for stance detection are not robust enough to be used in a viewpoint-diverse news recommender because the news constantly has new discussion topics. |
| Approach: | They propose to use two stance task definitions (Pro/Con versus Same Side Stance) and two LLM architectures (bi-encoding versus cross-encode) to test model performance. |
| Outcome: | The proposed models outperform the same side-stance definition and other models on stance across different topics. |
What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations (2023.findings-emnlp)
Copied to clipboard
Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, Yejin Choi
| Challenge: | Moral or ethical judgments rely heavily on the contexts in which they occur . a student model that produces defeasible contexts with improved validity, diversity, and defasibility is superior to intermediate student models . |
| Approach: | a new study uses a student model to provide contextualizations that make an action morally acceptable . the model is based on a dataset of 115K defeasible moral actions rated highly by human annotators . |
| Outcome: | The proposed model outperforms all intermediate models in a high-quality dataset . the model is based on 1.2M entries of contextualizations and rationales for 115K moral actions . |
One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot cross-lingual transfer are unreliable due to the lack of pretraining data. |
| Approach: | They propose to accumulatively average model snapshots from different runs into a single model. |
| Outcome: | The proposed protocol decouples performance maximization from hyperparameter tuning. |
IndoNLI: A Natural Language Inference Dataset for Indonesian (2021.emnlp-main)
Copied to clipboard
| Challenge: | XLM-R model outperforms other pre-trained models in annotated data. |
| Approach: | They adapt the data collection protocol for MNLI and collect 18K sentence pairs annotated by crowd workers and experts. |
| Outcome: | The proposed dataset outperforms other pre-trained models on the expert-annotated data. |
FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual models have been released, but many of the world's languages are not covered. |
| Approach: | They propose a method that initializes the embedding matrix for a new tokenizer based on information in the source model's embeddable matrix. |
| Outcome: | The proposed method outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). |
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text summarization have shown remarkable performance, but a significant number of summaries exhibit factual inconsistencies, such as hallucinations. |
| Approach: | They propose a factuality-oriented metric that evaluates text summarization for accuracy . they use a human annotation process to examine the accuracy of automatically generated summaries . |
| Outcome: | The proposed metric sets a new state-of-the-art on AGGREFACT, the de-facto benchmark for factuality evaluation. |
“Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation arises when multiple human annotators provide different labels for valid reasons. |
| Approach: | They propose to use crowd workers to represent human judgment distributions or expert linguists to provide detailed explanations for their chosen labels. |
| Outcome: | The proposed model can approximate human judgment distributions using a small number of expert labels and explanations. |
HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by looking at the hypothesis while completely ignoring the premise. |
| Approach: | They propose to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias. |
| Outcome: | The proposed models can be used to mitigate the hypothesis-only bias by using down-sampling and adversarial training. |
Automated Fact-Checking of Claims from Wikipedia (2020.lrec-1)
Copied to clipboard
| Challenge: | Fact checking datasets such as FEVER and SNLI suffer from limited applicability due to synthetic nature of claims and/or evidence written by annotators that differ from real claims and evidence on the internet. |
| Approach: | They present a dataset of 124k+ triples consisting of a claim, context and an evidence document extracted from English Wikipedia articles and citations. |
| Outcome: | The proposed dataset is the largest fact checking dataset consisting of real claims and evidence to date. |
Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on specialized BERT-variants and recent LLMs to reason inconsistencies. |
| Approach: | They propose to incorporate task-specific taxonomy into inferences to facilitate both zero-shot and supervised paradigms. |
| Outcome: | The proposed model outperforms specialized non-LLM and recent LLM models in a number of domains. |
Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that the interaction of large language models (LLMs) with theorem provers (TPs) can help verify and improve the validity of NLI explanations. |
| Approach: | They propose to use logical expressions to guide LLMs in generating structured proof sketches and to use them to improve their accuracy. |
| Outcome: | The proposed strategies improve autoformalisation, syntactic errors and explanation refinement over the state-of-the-art model. |
AXCEL: Automated eXplainable Consistency Evaluation using LLMs (2024.findings-emnlp)
Copied to clipboard
P Aditya Sreekar, Sahil Verma, Suransh Chopra, Abhishek Persad, Sarik Ghazarian, Narayanan Sadagopan
| Challenge: | Large Language Models (LLMs) are widely used for various tasks but evaluating the consistency of generated text remains a challenge. |
| Approach: | They propose a prompt-based consistency metric which provides explanations for consistency scores by providing detailed reasoning and pinpointing inconsistent text spans. |
| Outcome: | The proposed metric outperforms state-of-the-art metrics in summarization, free text generation and data-to-text conversion tasks by 8.7% and 6.2%. |
LogicAttack: Adversarial Attacks for Evaluating Logical Consistency of Natural Language Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated impressive performance on Natural Language Inference (NLI) tasks. |
| Approach: | They propose a method to attack NLI models using diverse logical forms of premise and hypothesis using propositional logic to generate effective adversarial attacks. |
| Outcome: | The proposed method achieves an average 53% Attack Success Rate (ASR) across multiple logic-based attacks. |
Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on metaphor processing have focused on single datasets and specific task settings, often using artificially constructed data through lexical replacement. |
| Approach: | They propose to evaluate the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations. |
| Outcome: | The proposed frameworks are more realistic and efficient than current models and are more efficient than existing models. |
Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions (2025.findings-emnlp)
Copied to clipboard
Meghdut Sengupta, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, Eyke Hüllermeier, Debanjan Ghosh, Henning Wachsmuth
| Challenge: | Metaphorical language is prevalent in everyday communication, often used unconsciously, as in "rising crime" incorporating conceptual metaphors improves performance on downstream tasks that implicitly require correct metaphor interpretation, such as NLI. |
| Approach: | They extend metaphorical texts in an existing NLI dataset by source and target domains and conduct an ablation study using Shapley values and interactions to assess the extent to which LLMs interpret metaphorical language correctly in NLI. |
| Outcome: | The proposed method improves performance on natural language inference tasks with metaphoric input. |
Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work on multilingual sentence embedding has demonstrated that the efficient use of natural language inference data to build high-performance models can outperform conventional methods. |
| Approach: | They propose a multilingual sentence embedding model by extending an existing monolingual model by using the low-rank adaptation technique. |
| Outcome: | The proposed model outperforms the previous approach and shows that languages with fewer resources or those with less linguistic similarity to English benefit more from the parameter increase. |
VerifyMatch: A Semi-Supervised Learning Paradigm for Natural Language Inference with Confidence-Aware MixUp (2024.emnlp-main)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is a key task for evaluating a model's ability to perform natural language understanding and reasoning. |
| Approach: | They propose to construct pseudo-generated samples using class-specific fine-tuned large language models (LLMs) . they retain all pseudo-labeled samples, but use MixUp to ensure unlabele . |
| Outcome: | The proposed approach achieves competitive accuracy compared to strong baselines for NLI datasets in low-resource settings. |
A MISMATCHED Benchmark for Scientific Natural Language Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for scientific NLI are derived from various computer science domains, whereas non-CS domains are completely ignored. |
| Approach: | They propose a scientific natural language inference benchmark called MisMatched that incorporates sentence pairs having an implicit scientific NLI relation into model training. |
| Outcome: | The proposed benchmark covers three non-CS domains and contains 2,700 human annotated sentence pairs. |
Rank-Awareness and Angular Constraints: A New Perspective on Learning Sentence Embeddings from NLI Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | High-quality sentence embeddings are critical for advancing a wide range of Natural Language Processing tasks. |
| Approach: | They propose a framework that leverages the full NLI dataset augmented with pre-computed continuous similarity scores (S) they employ a Rank Margin objective that enforces rank consistency against S using an explicit margin and a Gated Angular objective that conditionally refines embedding geometry based on NLI label (L) and S score agreement. |
| Outcome: | The proposed framework outperforms baseline models on STS and the MTEB benchmarks. |
Transitive self-consistency evaluation of NLI models without gold labels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that NLI models make inconsistent model decisions during inference . previous studies tested only variations of two transitive consistency rules out of four . |
| Approach: | They propose to generate adversarial examples using antonym replacements to test transitive consistency of NLI models. |
| Outcome: | The proposed approach can test all four rules automatically by generating adversarial examples via antonym replacements. |
Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to abstract inference ignore the *polysemy* and *hierarchical nature of concepts* . prevailing approaches disregard how arguments might entail differently across various concept levels, thereby missing potential enlargement connections. |
| Approach: | They propose a framework that organizes arguments hierarchically and delves into entailment relations at diverse concept levels. |
| Outcome: | The proposed framework improves the model's generalization and reasoning prowess in natural language inference. |
“You are Beautiful, Body Image Stereotypes are Ugly!” BIStereo: A Benchmark to Measure Body Image Stereotypes in Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | BIStereo is a suite of language models that uncover body image stereotypes in language models. |
| Approach: | They propose a metric, TriSentBias, that captures the biased preferences of LMs towards a certain body type over others. |
| Outcome: | The proposed metric captures biased preferences of LMs towards a certain body type over others. |
SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives (2024.lrec-main)
Copied to clipboard
| Challenge: | scalar implicatures are a phenomenon by which a speaker conveys the negation of a more informative utterance by producing a less informative . |
| Approach: | They propose to use a dataset to investigate the ability of language models to interpret utterances with scalar implicatures. |
| Outcome: | The proposed models perform significantly worse on in-domain and out-of-domain examples than other types of NLI examples. |
SI-NLI: A Slovene Natural Language Inference Dataset and Its Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for natural language inference (NLI) are limited to English and a few other well-resourced languages. |
| Approach: | They propose to use a dataset for natural language inference to extend the resources for the task. |
| Outcome: | The proposed dataset is constructed from scratch using knowledgeable annotators with carefully crafted guidelines aiming to avoid common problems in existing datasets. |
Dissecting Clinical Reasoning in Natural Language Inference for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities. |
| Approach: | They examine four classes of prompting strategies to elicit reasoning in large language models . they then construct demonstrations using a frontier model to distil multi-step reasoning capabilities into smaller models based on Low-Rank Adaptation (LoRA). |
| Outcome: | The proposed model improves in 75% of the models on MedNLI and TREC Clinical Trials. |
Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Standard language models employ unique, monolithic embeddings for each token, limiting their ability to capture multifaceted meanings. |
| Approach: | They propose a compositional structure that accumulates diverse semantic facets for tokens . they apply this representational scheme to standard transformer architectures and a biomedical domain benchmark . |
| Outcome: | The proposed representational scheme achieves extreme compression in embedding parameters while maintaining >95% task performance relative to the base model. |
Rule Discovery for Natural Language Inference Data Generation Using Out-of-Distribution Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training rules for natural language inference do not cover the diversity of natural language. |
| Approach: | They propose a framework that combines out-of-distribution detection and clustering to identify new premise–hypothesis pairs in a dataset that are not covered by existing rules. |
| Outcome: | The proposed framework achieves +0.85%p accuracy on 2k and +0.15%p on 550k samples. |
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations (2026.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often exhibit label variation. |
| Approach: | They extend LiTEx taxonomy to two NLI datasets and jointly analyze label variation and label variation. |
| Outcome: | The proposed model combines explanations as a lens to analyze variation in NLI annotations and examine individual differences in reasoning. |
Incorporating Temporal Coherence to Cross-Document Event Coreference Resolution (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches focus on enhancing semantic coherence between event mentions, but they overlook the critical aspect of temporal coherency. |
| Approach: | They propose a Temporal Cohorence-driven event coreference framework that explicitly models temporal constraints by constructing a temporal event graph and a GNN to resolve conflicts. |
| Outcome: | Experiments on the ECB+, GVC, WEC, and ECb+META datasets show that CohTP outperforms state-of-the-art methods. |
To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)
Copied to clipboard
| Challenge: | State-of-the-art language models perform well on a variety of language tasks, but struggle with understanding negation cues in tasks like natural language inference (NLI). |
| Approach: | They propose a new learning strategy for negation building on ELECTRA’s replaced token detection objective. |
| Outcome: | The proposed approach leads to substantial gains on a variant of RTE with additional negation. |
Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for presupposition generation fail to generate complete lists of presuffpositions. |
| Approach: | They propose to fine-tune existing BERT and T5 models for a task where a model produces a list of presuppositions carried by the given input sentence. |
| Outcome: | The proposed models outperform BERT and T5 models on the novel task of presupposition as natural language inference (PNLI) despite limited data, they achieved an emerging proficiency in generation of presumptions reaching ROUGE scores of 43.47, adhering to systematic patterns that mirror valid strategies for pres upposition generation, although failed to generate the complete lists. |
Entailed Between the Lines: Incorporating Implication into NLI (2025.acl-long)
Copied to clipboard
Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, Alex Fabrikant
| Challenge: | True Emotions, social cues, insults, and a myriad of other messages are conveyed implicitly, often even more so than explicitly. |
| Approach: | They propose a dataset to help LLMs understand implied entailment . |
| Outcome: | The proposed dataset enables LLMs to understand implied entailment and can generalize this understanding across datasets and domains. |
Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks (2025.acl-long)
Copied to clipboard
| Challenge: | Traditional methods that rely on 1:1 pairwise comparisons fail to capture inconsistencies . few studies have addressed related problems in the domain of factual inconsistenency detection . |
| Approach: | They propose a set-consistency verification task that assesses logical coherence of entire sets . they propose 'set-consistent energy network' that employs a margin-based loss to learn the compatibility among a collection of statements . |
| Outcome: | The proposed model outperforms existing methods and significantly outperformed existing models. |
Cross-Document Cross-Lingual NLI via RST-Enhanced Graph Fusion and Interpretability Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite the development of many subdirections, Cross-Document Cross-Lingual NLI remains largely unexplored. |
| Approach: | They propose a novel paradigm that extends traditional NLI capabilities to multi-document, multilingual scenarios by integrating RST-enhanced graph fusion with interpretability-aware prediction. |
| Outcome: | The proposed method improves on existing models and document-level NLI to multi-document, multilingual scenarios. |
Extractive Fact Decomposition for Interpretable Natural Language Inference in one Forward Pass (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in Natural Language Inference (NLI) uses atomic fact decomposition to enhance interpretability and robustness. |
| Approach: | They propose an encoder-only architecture that performs extractive atomic fact decomposition and interpretable inference without generative models. |
| Outcome: | The proposed architecture achieves competitive accuracy and improves robustness out of distribution and in adversarial settings over models based on extractive rationale supervision. |
WojoodRelations: Arabic Relation Extraction Corpus and Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on Arabic RE remains limited due to the language’s rich morphology and syntactic complexity, and the lack of large, high-quality datasets. |
| Approach: | They propose to use WojoodRelations to extract relation relationships from Arabic textual data using relation-aware templates and GPT-Joint to perform relation-based retrieval. |
| Outcome: | The proposed method achieves a Cohen’s of 0.92, indicating high reliability, and supervised models achieve 92.89% F1 for RE, while LLMs obtain 72.73% F1 . |
Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce incomplete or selectively omit key information . omissions of key information or misrepresentation of conflicting evidence can cause harm . |
| Approach: | They propose a method that decomposes texts into atomic statements and uses natural language inference to identify missing facts and a Q A-based metric that extracts question-answer pairs and compares responses across sources. |
| Outcome: | The proposed evaluation metrics show they perform better than more complex metrics, but at a cost. |