Parameter-efficient Modularised Bias Mitigation via AdapterFusion (2023.eacl-main)
Copied to clipboard
Deepak Kumar, Oleg Lesota, George Zerveas, Daniel Cohen, Carsten Eickhoff, Markus Schedl, Navid Rekabsaz
| Challenge: | Large pre-trained language models contain societal biases and carry along these biase . Current approaches to mitigate these bias impose debiasing by updating model parameters, effectively transferring model to irreversible debiased state. |
| Approach: | They propose to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand while keeping the core model untouched. |
| Outcome: | The proposed approach improves or maintains effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance while granting parameter-efficiency and easy switching between the original and debiased models. |
Similar Papers
Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained language models can be used to mitigate societal biases and stereotypes. |
| Approach: | They propose a modular bias mitigation approach that integrates debiasing modules into the core model on-demand at inference time. |
| Outcome: | The proposed approach improves on-par with baseline finetuning on gender, race, and age protected attributes on three classification tasks with gender, age, and race as protected attributes. |
PEFTDebias : Capturing debiasing information using PEFTs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research suggests that biases acquired during pretraining can propagate to downstream models, resulting in superficial text dependencies and potential implicit bias. |
| Approach: | They propose a parameter-efficient fine-tuning approach to mitigate implicit biases within foundation models by incorporating parameters into the model and freezing them during the fine-uning process. |
| Outcome: | The proposed method reduces biases in foundation models by incorporating parameters and freezing them during fine-tuning. |
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021.naacl-main)
Copied to clipboard
| Challenge: | PTLMs can exhibit biases against protected groups in a host of modeling tasks . but, fine-tuned LMs may propagate bias to downstream classifiers . |
| Approach: | They propose to use upstream bias mitigation techniques to reduce bias on downstream tasks by fine-tuning an upstream model and applying it to a downstream model. |
| Outcome: | The proposed model reduces bias on hate speech detection, toxicity detection and coreference resolution tasks over bias factors. |
Effective Controllable Bias Mitigation for Classification and Retrieval using Gate Adapters (2024.eacl-long)
Copied to clipboard
| Challenge: | a recent study focused on learning separate modules for on-demand debiasing. |
| Approach: | They propose a modular debiasing module with a controllable gate adapter . they demonstrate that the module can reduce the bias of search results . |
| Outcome: | The proposed module can reduce biases on three classification tasks with four protected attributes while maintaining higher task performance. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |
Sustainable Modular Debiasing of Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing debiasing methods modify all of the PLM parameters, which is costly and leads to (catastrophic) forgetting of useful language knowledge. |
| Approach: | They propose a modular debiasing approach based on dedicated adapters that inject adapter modules into the original PLM layers and update only the adapters. |
| Outcome: | The proposed approach is based on dedicated adapters and retains fairness even after large-scale training. |
Open-DeBias: Toward Mitigating Open-Set Bias in Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to addressing harmful biases in LLMs are limited to predefined categories . a novel, data-efficient, and parameter-efficient debiasing method is proposed to mitigate existing social and stereotypical biase . |
| Approach: | They propose an open-set bias detection and mitigation method to address harmful biases in text-based QA. |
| Outcome: | The proposed method improves QA accuracy on Korean BBQ dataset by nearly 48% on ambiguous subsets and 6% on disambiguated ones. |
Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models can encode unfair social biases from large pre-training corpora and even amplify biase in downstream applications. |
| Approach: | They propose a *debias-while-prompt tuning* method for mitigating biases via counterfactual contrastive prompt tuning on downstream tasks. |
| Outcome: | The proposed method can mitigate biases on three extrinsic bias benchmarks and adapt to existing debiased language models. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
How Far Can It Go? On Intrinsic Gender Bias Mitigation for Text Classification (2023.eacl-main)
Copied to clipboard
| Challenge: | a growing interest in exploring how gender bias pertains in contextualized language models has been generated . intrinsic mitigation strategies and bias metrics have been proposed to mitigate gender bias in contextualised language models . |
| Approach: | They propose to use different intrinsic bias mitigation strategies to mitigate gender bias in contextualized language models. |
| Outcome: | The proposed probe shows that some mitigation techniques can hide gender bias . the probe also shows that not all mitigation techniques fool extrinsic bias despite their use . |