Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions (2023.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to de-bias pre-trained large language models focus on changes to training regime, but this is not feasible. |
| Approach: | They propose to de-bias a pre-trained model by fine-tuning it on only 10 examples . they show that the technique performs better than competitive baselines . |
| Outcome: | The proposed method performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability. |
Similar Papers
Leveraging Pre-trained Language Models for Gender Debiasing (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to reduce gender bias in natural language are costly and time-consuming. |
| Approach: | They propose a method to generate gender variants for a given text using pre-trained language models as the resource without any task-specific labelled data. |
| Outcome: | The proposed method can reduce gender bias in a language generation context without a task-specific labelled data. |
Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for debiasing are resource-intensive and costly. Existing solutions for debiansing require fine-tuning on downstream tasks. |
| Approach: | They propose to integrate Masked Language Modeling (MLM) training objectives into fine-tuning’s training process to debiase the PLMs. |
| Outcome: | The proposed approach outperforms the state-of-the-art baselines in terms of gender bias scores while improving PLMs’ performance solely using the downstream tasks’ dataset. |
Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function (P19-2)
Copied to clipboard
| Challenge: | Existing methods to reduce gender bias in natural language datasets are inadequate. |
| Approach: | They propose a loss function modification approach which equalizes the probabilities of male and female words in the output. |
| Outcome: | The proposed approach outperforms existing methods in several aspects, especially in reducing gender bias in occupation words. |
Reducing Gender Bias in Abusive Language Detection (D18-1)
Copied to clipboard
| Challenge: | Abusive language detection models tend to be biased toward identity words of a certain group of people . recent studies have raised concerns about the robustness of such systems . |
| Approach: | They propose to use debiased word embeddings, gender swap data augmentation to reduce model bias . they also propose to fine-tune models with a larger corpus to correct such bias if needed . |
| Outcome: | The proposed methods reduce model bias by 90-98% and can be extended to correct model bias in other scenarios. |
In-Contextual Gender Bias Suppression for Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally expensive. |
| Approach: | They propose to suppress gender biases by providing textual preambles from manually designed templates and real-world statistics without accessing model parameters. |
| Outcome: | The proposed methods suppress gender biases in English LLMs using a CrowsPairs dataset without accessing model parameters. |
Identifying and Reducing Gender Bias in Word-Level Language Models (N19-3)
Copied to clipboard
| Challenge: | Existing discriminatory biases in training data can be amplified by models . text corpora exhibit socially problematic biase . |
| Approach: | They propose a metric to measure gender bias and a regularization loss term to minimize embeddings onto an embeddable subspace that encodes gender. |
| Outcome: | The proposed method reduces gender bias up to an optimal weight assigned to the loss term, and the model becomes unstable as the perplexity increases. |
On Evaluating and Mitigating Gender Biases in Multilingual Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks and resources for evaluating gender biases in multilingual settings are limited. |
| Approach: | They propose to extend DisCo to different Indian languages using human annotations to evaluate gender biases in multilingual models. |
| Outcome: | The proposed benchmarks and mitigation techniques are extended beyond English to evaluate gender biases in multilingual models. |
Gender-preserving Debiasing for Pre-trained Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods for debiasing word embeddings have shown discriminative biases . word embeds learnt from social media have shown to encode racist, offensive and discriminative language usage. |
| Approach: | They propose a method that preserves gender-related information while removing stereotypical gender biases from pre-trained word embeddings. |
| Outcome: | The proposed method preserves gender-related information while removing stereotypical discriminative gender biases from pre-trained word embeddings. |
Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work has explored using sequence-to-sequence rewriting models to transform biased outputs into more gender-fair language by creating pseudo training data through linguistic rules. |
| Approach: | They propose to use machine translation models to create gender-biased text from real gender-fair text via round-trip translation to eliminate rule-based data creation. |
| Outcome: | The proposed approach matches the performance of state-of-the-art rewriting models for English. |
An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models inherit more human-like biases from the training corpora, causing computationally expensive problems. |
| Approach: | They propose parameter-efficient methods in combination with counterfactual data augmentation for bias mitigation. |
| Outcome: | The proposed methods are effective in mitigating gender bias, prompt tuning is more suitable for GPT-2 than BERT, and less effective when it comes to racial and religious bias. |