Papers by Joe Stacey
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI. |
| Approach: | They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases. |
| Outcome: | The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved . |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
Atomic Inference for NLI with Generated Facts as Atoms (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that can provide accurate explanations are not interpretable, i.e. they do not reflect the inner workings of the model. |
| Approach: | They propose to use LLM-generated facts as atoms to make interpretable models that can be used to make accurate predictions for each component part of an input. |
| Outcome: | The proposed method outperforms existing methods on natural language understanding tasks with a multi-stage fact generation process and a training regime that incorporates the facts. |
Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural models pick up on annotation artefacts and spurious correlations, resulting in learning sentences that suffer from the same biases. |
| Approach: | They propose to tackle this problem by using adversarial training to reduce the bias in sentence representations by using an ensemble of adversaries. |
| Outcome: | The proposed approach produces more robust models outperforming previous de-biasing efforts when generalised to 12 other NLI datasets. |
AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios (2026.acl-long)
Copied to clipboard
| Challenge: | brittleness of Large Language Models in reasoningintensive tasks is a problem . current compositional benchmarks focus on *either* commonsense or math reasoning . |
| Approach: | They propose a "Co**mmonsense and Ma**th" benchmark where each compositional task requires a commonsense reasoning step *and* a math reasoning step. |
| Outcome: | The proposed benchmarks show that LLMs can solve both steps in isolation, but their accuracy drops by nearly 30% when the two steps are combined. |
LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues (2024.naacl-srw)
Copied to clipboard
Joe Stacey, Jianpeng Cheng, John Torr, Tristan Guigue, Joris Driesen, Alexandru Coca, Mark Gaynor, Anders Johannsen
| Challenge: | Existing datasets with limited domain coverage and few challenging conversational phenomena are often unlabelled . Existing data is limited in quality and lacks a robust evaluation process . |
| Approach: | They propose a high quality data generation system that generates high quality dialogues using 4,277 conversations across 100 intents. |
| Outcome: | The proposed system produces high quality dialogue data with high quality labels. |