MNLI

Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning (2026.acl-srw)

Challenge:	In this paper, we examine the ability of large language models (LLMs) to accommodate different pragmatic sentence types, such as questions, commands, and sentence fragments for natural language inference (NLI).
Approach:	They propose to fine-tune large language models to accommodate different sentence types for natural language inference (NLI) they also explore ChatGPT's concept of entailment by using a symbolic semantic parser.
Outcome:	The proposed models can accommodate different sentence types without losing too much accuracy on MNLI-matched models.

Copied to clipboard

Brady Steele

Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (D19-61)

Challenge:	Annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity, but most fine-tuning recipes treat every example as equallylearnable.
Approach:	They ask whether annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity.
Outcome:	The proposed method predicts learning dynamics on contested examples under LoRA.

Copied to clipboard

He He, Sheng Zha, Haohan Wang

Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning (2021.findings-acl)

Challenge:	Statistical natural language inference models are susceptible to learning dataset bias.
Approach:	They propose a debiasing algorithm that debiases models that use only known dataset biases . they use two benchmark datasets to train three high-performing NLI models .
Outcome:	The proposed learning objective improves model performance on challenge datasets while maintaining reasonable performance on original datasets.

Copied to clipboard

Benjamin Minixhofer, Milan Gritta, Ignacio Iacobacci

English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020.aacl-main)

Challenge:	Recent advances in transfer learning have brought significant improvements to many natural language processing tasks.
Approach:	They propose a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network.
Outcome:	The proposed method improves on several NLI datasets using a strong baseline model (RoBERTa-large) with MNLI pretraining.

Copied to clipboard

Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Samuel R. Bowman

Measuring and Mitigating Shortcut Reliance in Language Models with Probe-Based Representation Entanglement (2026.acl-srw)

Challenge:	a study of intermediate-task training in monolingual English shows that it improves model performance on non-English language understanding tasks.
Approach:	They evaluate whether English intermediate-task training is still helpful on non-English target tasks . BUCC and Tatoeba sentence retrieval tasks see large improvements .
Outcome:	The proposed model outperforms existing models on non-English language understanding tasks.

Copied to clipboard

Divyajot Singh

CondenseLM: LLMs-driven Text Dataset Condensation via Reward Matching (2025.emnlp-main)

Challenge:	Shortcut learning remains a major obstacle to robust NLP systems.
Approach:	They propose to fine-tune Gemma 3 1B Instruct and Llama 3.2 1B on two synthetic sentiment shortcuts in SST-2 and one natural shortcut in MNLI based on lexical overlap.
Outcome:	The proposed model improves on two synthetic sentiment shortcuts and one natural shortcut in MNLI with a 99% shortcut ratio, while Gemma drops from 91.8% to 60.2%.

Copied to clipboard

Cheng Shen, Yew-Soon Ong, Joey Tianyi Zhou

Green & Sustainable NLP

Challenge:	Existing methods for dataset condensation struggle to compress more information into samples . current methods struggle to extract enough training data for effective learning .
Approach:	They propose a paradigm for dataset condensation that uses an LLMs-driven approach to generate more informative and less biased samples.
Outcome:	The proposed method outperforms coreset selection and existing condensation methods by large margins while significantly reducing the computational cost.

Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020.aacl-main)

Copied to clipboard

Clara Vania, Ruijie Chen, Samuel R. Bowman

Benchmark Transparency: Measuring the Impact of Data on Evaluation (2024.naacl-long)

Challenge:	Large-scale natural language inference datasets are available for non-expert crowdsourcing.
Approach:	They propose to use crowdworker writing to create candidate pairs for annotation . they find crowdworkers writing is the best known protocol for entailment data .
Outcome:	The proposed protocol outperforms the baseline protocol in evaluations of generalization and transfer learning.

Copied to clipboard

Venelin Kovatchev, Matthew Lease

Comparing Test Sets with Item Response Theory (2021.acl-long)

Challenge:	In this paper, we quantify the impact that data distribution has on the performance and evaluation of NLP models.
Approach:	They propose to use disproportional stratified sampling to measure the data distribution across 6 different dimensions to quantify model performance.
Outcome:	The proposed framework measures the data distribution across 6 different dimensions and shows that it is statistically significant and predicts model performance.

Copied to clipboard

Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman

Linguistic Theories

Challenge:	Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress.
Approach:	They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples.
Outcome:	The proposed datasets are saturated and unlikely to detect future improvements.

Outlier Dimensions that Disrupt Transformers are Driven by Frequency (2022.findings-emnlp)

Copied to clipboard

Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, Felice Dell’Orletta

Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)

Challenge:	Disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI.
Approach:	They find that disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI.
Outcome:	The proposed model outlier phenomenon is associated with the frequency of encoded tokens in pre-training data.

Copied to clipboard

Houjun Liu, John Bauer, Christopher D Manning

MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (2022.naacl-main)

Challenge:	Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit.
Approach:	They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance.
Outcome:	The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI.

Copied to clipboard

Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, Weizhu Chen

VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)

Challenge:	Existing methods for training pre-trained language models have limited practicality due to latency requirements.
Approach:	They propose a method that uses a Mixture-of-Experts structure to increase model capacity and inference speed.
Outcome:	The proposed method outperforms existing distillation methods on natural language understanding and question answering tasks.

Copied to clipboard

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, Barbara Plank

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)

Challenge:	Existing work on label variation and annotation errors has focused on them in isolation.
Approach:	They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations.
Outcome:	The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items.

Copied to clipboard

Joe Stacey, Marek Rei

NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)

Challenge:	Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits.
Approach:	They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target.
Outcome:	The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain.

Copied to clipboard

Haoxin Li, Phillip Keung, Daniel Cheng, Jungo Kasai, Noah A. Smith

UDAPTER - Efficient Domain Adaptation Using Adapters (2023.eacl-main)

Challenge:	Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time.
Approach:	They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x.
Outcome:	The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation.

Copied to clipboard

Bhavitvya Malik, Abhinav Ramesh Kashyap, Min-Yen Kan, Soujanya Poria

Are Prompt-based Models Clueless? (2022.acl-long)

Challenge:	Using adapters, unsupervised domain adaptation (UDA) is more parameter efficient and requires large-scale data to be effective.
Approach:	They propose to add small bottleneck layers to each layer of a pre-trained language model to make it more parameter efficient by adding adapters.
Outcome:	The proposed methods outperform unsupervised domain adaptation methods such as DANN and DSN in natural language inference and sentiment classification tasks.

Copied to clipboard

Pride Kavumba, Ryo Takahashi, Yusuke Oda

Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)

Challenge:	Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective.
Approach:	They propose to examine whether few-shot prompt-based models exploit superficial cues by reusing the model head and formatting the input to match the pre-training objective.
Outcome:	The proposed models perform well on instances with superficial cues, but often outperform random accuracy on instances without superficial cuing.

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Marek Rei

Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)

Challenge:	Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI.
Approach:	They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases.
Outcome:	The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved .

Copied to clipboard

Cheng Luo, Wei Liu, Jieyu Lin, Jiajie Zou, Ming Xiang, Nai Ding

Increasing Robustness to Spurious Correlations using Forgettable Examples (2021.eacl-main)

Challenge:	Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say .
Approach:	They propose to use syntactically simple sentences to test the inference ability of NLI models.
Outcome:	The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair.

Copied to clipboard

Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, T. J. Hazen, Alessandro Sordoni

Robustness in NLP

Challenge:	Neural NLP models often exploit spurious correlations to perform tasks. minority examples have been shown to increase the out-of-distribution generalization of pre-trained language models.
Approach:	They propose to use example forgetting to find minority examples without prior knowledge of spurious correlations in the dataset.
Outcome:	The proposed approach improves out-of-distribution generalization on minorities . it shows that minority examples are more robust on challenging datasets .

OCNLI: Original Chinese Natural Language Inference (2020.findings-emnlp)

Copied to clipboard

Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, Lawrence Moss

Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level (2021.findings-acl)

Challenge:	Recent efforts to extend natural language understanding to other languages have focused on (automatically) translating existing English datasets.
Approach:	They propose to use a Chinese dataset to generate annotated sentences from native speakers specializing in linguistics to elicit annotations.
Outcome:	The proposed dataset does not rely on automatic translation or non-expert annotation. instead, it elicits annotations from native speakers specializing in linguistics.

Copied to clipboard

Ruiqi Zhong, Dhruba Ghosh, Dan Klein, Jacob Steinhardt

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (P19-1)

Challenge:	Larger models have higher out-of-distribution robustness, while smaller models have lower accuracy on rare subgroups.
Approach:	They develop statistically rigorous methods to investigate whether large models are better on every instance . they find that individual predictions are highly sensitive to noise in the randomness in training .
Outcome:	The proposed model is worse than BERT-MINI on 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%.

Copied to clipboard

Tom McCoy, Ellie Pavlick, Tal Linzen

How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks (2021.findings-emnlp)

Challenge:	lexical overlap heuristics are effective for frequent example types but break down in more challenging cases.
Approach:	They propose to use a set of examples to test whether a sentence entails another . they propose to adopt three fallible syntactic heuristics for statistical NLI models .
Outcome:	The proposed model performs poorly on the HANS dataset, suggesting they have adopted these heuristics.

Copied to clipboard

Hoang Van, Zheng Tang, Mihai Surdeanu

Don’t Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques (2021.findings-emnlp)

Challenge:	Recent studies have focused on rule-based and neural sequence-to-sequence (seq2sequ) TS is a technique that reduces text complexity for human consumption.
Approach:	They evaluate two possible uses of neural TS: simplifying input texts at prediction time and augmenting training data to provide machines with additional information during training.
Outcome:	The proposed approach improves performance on two datasets.

Copied to clipboard

Hossein Amirkhani, Mohammad Taher Pilehvar

Rule Augmented Unsupervised Constituency Parsing (2021.findings-acl)

Challenge:	Existing techniques for mitigating dataset bias often leverage a biased model to identify biased instances. Existing methods for mitiging dataset bias use shallow patterns that can be exploited by the model.
Approach:	They propose to use partial-input and limited-capacity models to detect biased instances and reduce their role during training to enhance its robustness to out-of-distribution data.
Outcome:	The proposed method outperforms existing methods for mitigating dataset bias on two well-known datasets in the domain, MNLI and FEVER.

Copied to clipboard

Atul Sahay, Anshul Nasery, Ayush Maheshwari, Ganesh Ramakrishnan, Rishabh Iyer

Ecologically Valid Explanations for Label Variation in NLI (2023.findings-emnlp)

Challenge:	Recent studies have shown that unsupervised parsing methods do not learn meaningful semantics (not even simple grammar)
Approach:	They propose an approach that utilizes very generic linguistic knowledge of the language present in the form of syntactic grammar rules and is independent of the base system.
Outcome:	The proposed model is independent of the base system and takes advantage of syntactic grammar rules.

Copied to clipboard

Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe

What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)

Challenge:	Human label variation exists in many natural language processing tasks, including NLI .
Approach:	They build an English dataset of 1,415 ecologically valid explanations for 122 MNLI items . they find that people can systematically vary on their interpretation .
Outcome:	The proposed dataset contains 1,415 ecologically valid explanations for 122 items . the results show that people can vary on interpretation and highlight differences .

Copied to clipboard

Yixin Nie, Xiang Zhou, Mohit Bansal

IndoNLI: A Natural Language Inference Dataset for Indonesian (2021.emnlp-main)

Challenge:	Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions.
Approach:	They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels .
Outcome:	The proposed dataset examines the distribution of human opinions in NLU evaluation datasets.

Copied to clipboard

Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, Clara Vania

C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs (2026.findings-acl)

Challenge:	XLM-R model outperforms other pre-trained models in annotated data.
Approach:	They adapt the data collection protocol for MNLI and collect 18K sentence pairs annotated by crowd workers and experts.
Outcome:	The proposed dataset outperforms other pre-trained models on the expert-annotated data.

Copied to clipboard

Xuan Feng, Bo An, Tianlong Gu, Liang Chang, Fengrui Hao, Peipeng Yu, Shuai Zhao

To Learn or Not to Learn: Replaced Token Detection for Learning the Meaning of Negation (2024.lrec-main)

Challenge:	Existing approaches to solve large language models address stereotypical and structural biases in isolation . however, prior paradigms address these in isolation, often at the expense of exacerbating the other .
Approach:	They propose a framework to tackle latent spurious feature correlations within input that drive erroneous reasoning shortcuts.
Outcome:	The proposed framework mitigates stereotypical and structural biases while preserving robust general reasoning capabilities.

Copied to clipboard

Gunjan Bhattarai, Katrin Erk