An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on. |
| Approach: | They propose to use Counterfactual Data Augmentation, Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia as bias mitigation techniques to quantify their effectiveness. |
| Outcome: | The proposed techniques are Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebia. |
Similar Papers
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning (2022.aacl-main)
Copied to clipboard
| Challenge: | Large-scale, pretrained vision-language models are growing in popularity due to impressive performance on downstream tasks with minimal finetuning. |
| Approach: | They propose to apply ranking metrics to image-text representations to investigate bias measures and debiasing methods to reduce various bias measures. |
| Outcome: | The proposed model reduces bias measures with minimal degradation to image-text representations. |
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has focused on measuring and mitigating bias in pretrained language models. |
| Approach: | They propose a dataset that measures and mitigates bias across gender,race, religion, and queerness . they compare REDDITBIAS to a widely used conversational DialoGPT model . |
| Outcome: | The proposed framework measures and mitigates bias across gender,race, religion, and queerness dimensions. |
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on social biases in language models have focused on only English. |
| Approach: | They propose to use a Chinese dataset for bias evaluation and mitigation of Chinese conversational language models. |
| Outcome: | The proposed dataset includes under-explored bias categories, such as ageism and appearance biases, which received less attention in previous studies. |
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)
Copied to clipboard
| Challenge: | Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet. |
| Approach: | They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. |
| Outcome: | The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior. |
End-to-End Bias Mitigation by Modelling Biases in Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that strong natural language understanding models are prone to relying on unwanted dataset biases without learning the underlying task. |
| Approach: | They propose two learning strategies to train neural models that are more robust to dataset biases and transfer better to out-of-domain datasets. |
| Outcome: | The proposed methods improve robustness in all settings and transfer better to out-of-domain datasets. |
Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detection of biases in contextual language models are inconsistent and inconclusive. |
| Approach: | They propose to use word embedding association test to detect biases in contextual language models to compare them with other methods. |
| Outcome: | The proposed methods are inconsistent and inconclusive for language models with word embeddings. |
Data-Centric Explainable Debiasing for Improving Fairness in Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing data-centric debiasing strategies mainly leverage explicit bias words for counterfactual data augmentation to balance the training data. |
| Approach: | They propose a method which uses an explainability method to search for implicit bias words to assist in debiasing PLMs. |
| Outcome: | Extensive results show that the proposed method achieves state-of-the-art debiasing performance and strong generalization while maintaining predictive abilities. |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
Projective Methods for Mitigating Gender Bias in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Mitigating gender bias in NLP has a long history tied to debiasing static word embeddings. |
| Approach: | They propose a masked language modelling task where content is developed around known social stereotypes and a projective debiasing method is used to reduce bias. |
| Outcome: | The proposed methods reduce intrinsic bias and mitigat observed bias in a downstream setting, but the two outcomes are not necessarily correlated. |