Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
Similar Papers
Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models can encode unfair social biases from large pre-training corpora and even amplify biase in downstream applications. |
| Approach: | They propose a *debias-while-prompt tuning* method for mitigating biases via counterfactual contrastive prompt tuning on downstream tasks. |
| Outcome: | The proposed method can mitigate biases on three extrinsic bias benchmarks and adapt to existing debiased language models. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)
Copied to clipboard
| Challenge: | Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet. |
| Approach: | They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. |
| Outcome: | The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior. |
UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts (2024.eacl-srw)
Copied to clipboard
| Challenge: | Language models (LMs) often include societal biases encoded in the human-produced datasets used for their training. |
| Approach: | They evaluated six prominent language models: BERT, RoBERTa, DistilBERT, BERT- multilingual, XLM-RoBERT and DistilberT- multilinguistic. |
| Outcome: | The results show that the models generated by the models were stereotypically gendered and with a reduced bias in multilingual variants. |
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on. |
| Approach: | They propose to use Counterfactual Data Augmentation, Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia as bias mitigation techniques to quantify their effectiveness. |
| Outcome: | The proposed techniques are Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia. |
Mitigating Biases in Language Models via Bias Unlearning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent debiasing approaches target different demographic groups, harming fairness and discrimination. |
| Approach: | They propose a model debiasing framework which targets stereotypes by unlearning stereotype forgetting and anti-stereotype retention. |
| Outcome: | The proposed framework outperforms existing methods in mitigating bias while retaining language modeling capabilities. |
A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning (2022.aacl-main)
Copied to clipboard
| Challenge: | Large-scale, pretrained vision-language models are growing in popularity due to impressive performance on downstream tasks with minimal finetuning. |
| Approach: | They propose to apply ranking metrics to image-text representations to investigate bias measures and debiasing methods to reduce various bias measures. |
| Outcome: | The proposed model reduces bias measures with minimal degradation to image-text representations. |
Debiasing Large Language Models with Structured Knowledge (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce biases in pre-training models are hampered by their performance. |
| Approach: | They propose a method that utilizes structured knowledge to mitigate bias in LLMs . their method obviates the need for training from scratch, thus offering enhanced scalability . |
| Outcome: | The proposed method outperforms state-of-the-art (SOTA) baselines in the debiasing ability. |
Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes (2025.naacl-short)
Copied to clipboard
Isabel O. Gallegos, Ryan Aponte, Ryan A. Rossi, Joe Barrow, Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, Franck Dernoncourt, Nedim Lipka, Deonna Owens, Jiuxiang Gu
| Challenge: | Large language models exhibit harmful social biases, but they are often difficult to train and modify. |
| Approach: | They leverage the zero-shot capabilities of large language models to reduce stereotyping . they introduce a technique called zero- shot self-debiasing to reduce bias . |
| Outcome: | The proposed technique reduces stereotyping across nine different social groups while relying on the LLM itself and a simple prompt. |
Causal-Debias: Unifying Debiasing in Pretrained Language Models and Fine-tuning via Causal Invariant Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to remove unwanted stereotypical associations from pretrained language models (PLMs) are often focused on removing unwanted stereotypes from PLMs. |
| Approach: | They propose a framework to remove unwanted stereotypical associations in pretrained language models . they propose bias-relevant factors are causal, while labelrelevant factors causal . |
| Outcome: | The proposed framework reduces stereotypical associations after PLMs are fine-tuned . the proposed framework mitigates bias from a causal invariant perspective . |