| Challenge: | Existing models fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. |
| Approach: | They propose a framework that automatically identifies challenging subgroups and generates new data for those subgroup using large language models with a human in the loop. |
| Outcome: | The proposed framework improves accuracy on challenging subgroups while improving overall test accuracy. |
Similar Papers
People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Past work has shown that counterfactually augmented data (CADs) can improve models' performance on out-of-domain tests. |
| Approach: | They use Polyjuice, ChatGPT, and Flan-T5 to automatically generate CADs . they find that CAD generates a model that flips the original label with minimal changes . |
| Outcome: | The proposed model improves model robustness on out-of-domain test sets and individual data points. |
CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial text generation approaches can lead to generation lacking diversity or fluency, whereas perturbing in the intermediate representation space can lead a model to generate generations that are not related to the input. |
| Approach: | They propose to generate adversarial texts through controllable attributes that are known to be invariant to task labels. |
| Outcome: | The proposed model generates more diverse and fluent adversarial examples, compared to existing approaches, and is more robust against model re-training and different model architectures. |
Towards Universal Debiasing for Language Models-based Tabular Data Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large language models have exacerbated fairness issues in tabular data generation . inherent historical biases in tabulated data cause LLMs to exacerbate fairness problems . |
| Approach: | They propose a universal debiasing framework that minimizes group-level dependencies . it leverages the autoregressive structure and analytic sampling distributions of LLM-based tabular data generators . |
| Outcome: | The proposed framework minimizes group-level dependencies while reducing mutual information between advantaged and protected attributes. |
CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | a low-resource dataset is limited in training data, so generating task-specific data is challenging. |
| Approach: | They propose a data augmentation technique that prompts off-the-shelf instruction-following Large Language Models to generate augmentations. |
| Outcome: | The proposed technique outperforms baselines on 11 datasets spanning 3 tasks and 3 low-resource settings. |
Attack Prompt Generation for Red Teaming and Defending Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies construct attack prompts via manual or automatic methods, but these methods have limitations on cost and quality. |
| Approach: | They propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning and a defense framework that fine-tunes victim LLM's through iterative interactions with the attack framework. |
| Outcome: | The proposed approach is based on experiments on different LLMs to evaluate their effectiveness against red teaming attacks. |
Prediction-Augmented Generation for Automatic Diagnosis Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) adopt autoregressive architecture, predicting the next word token based on the preceding context. |
| Approach: | They propose a method that integrates task-specific predictive models as external tools to improve model generation quality and accuracy. |
| Outcome: | The proposed method improves the generation quality and predictive accuracy of large language models in inference-driven tasks. |
Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce model's reliance on bias features ignore the learnability of these features. |
| Approach: | They propose to reduce models' reliance on bias features by first training models with fixed low-capacity models which ignore the learnability of the bias features. |
| Outcome: | The proposed models can perform better on out-of-distribution datasets than baseline models with a more sophisticated model design. |
Better Synthetic Data by Retrieving and Transforming Existing Datasets (2024.findings-acl)
Copied to clipboard
| Challenge: | despite advances in large language models, task-specific data is not available for many use cases . a new method to improve automated dataset generation uses publicly available datasets . |
| Approach: | They propose a method to make better use of existing datasets to improve automatic dataset generation. |
| Outcome: | The proposed method outperforms existing methods on language-based tasks . it significantly increases diversity and difficulty of generated data on many tasks compared to other methods . |
Making Large Language Models Better Data Creators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced the field of NLP significantly, but deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. |
| Approach: | They propose a unified data creation pipeline that requires only a single formatting example. |
| Outcome: | The proposed pipeline can generate data with a single formatting example. |
Generation-Based Data Augmentation for Offensive Language Detection: Is It Worth It? (2023.eacl-main)
Copied to clipboard
| Challenge: | generative data augmentation has been shown to be effective in offensive language detection but the potential for bias injection has not been investigated. |
| Approach: | They propose to investigate the robustness of models trained on generated data in a variety of data augmentation setups and analyze models using the HateCheck suite. |
| Outcome: | The proposed model training setups on four English offensive language datasets are robust and robust, while the generative DA setups do not present bias injection issues. |