No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has relied on word embedding bias measures, such as WEAT, but these methods can be inaccurate due to several representation issues, such low-resource settings and token frequency differences. |
| Approach: | They propose to use WEAT to quantify social bias in US online news articles and embed embedding algorithms to account for the aforementioned issues. |
| Outcome: | The proposed algorithms do not match the literature, but they reduce the gap. |
Similar Papers
When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People? (2020.acl-main)
Copied to clipboard
| Challenge: | a study of word embeddings shows that social biases are more accurate than survey data for some dimensions of meaning. |
| Approach: | a new study investigates the extent to which word embeddings accurately reflect biases . they find that biased word embeds mirror survey data across 17 dimensions of social meaning . |
| Outcome: | a new study shows that word embeddings accurately reflect biases on average across dimensions of social meaning . biased embedders are more reflective of survey data for some dimensions of meaning than others, the study finds . |
Measuring Social Biases in Grounded Vision and Language Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to measure social biases in word embeddings are limited to visually grounded word embeds . a new study generalizes word embedment associations to visually ground word embeddas . |
| Approach: | They generalize word embeddings' biases to visually grounded word embeds . they propose two generalizations that answer questions about how biase, language, and vision interact . |
| Outcome: | The proposed measures are applied to a new dataset that includes 10,228 images from COCO, Conceptual Captions, and Google Images. |
SOS: Systematic Offensive Stereotyping Bias in Word Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Systematic Offensive Stereotyping (SOS) in word embeddings could lead to associating marginalised groups with hate speech and profanity. |
| Approach: | They propose a quantitative measure of the systematic offensive stereotyping (SOS) in word embeddings and validate it in most commonly used word embeds. |
| Outcome: | The proposed measure correlates with published statistics on online extremism, but does not explain hate speech detection models. |
Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies have evaluated social biases in word embeddings, but they are understudied. |
| Approach: | They propose to evaluate the social biases in sense embeddings using a benchmark dataset for word embedders. |
| Outcome: | The proposed measures show that even when no biases are found at word-level, there are still worrying levels of social biase at sense-level which are often ignored by the word- level bias evaluation measures. |
Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to quantify gender bias in word embeddings are not robust and cannot identify common types of bias. |
| Approach: | They propose to quantify gender bias by using cosine similarity to a pair of gender words and using analogies. |
| Outcome: | The proposed methods are not robust and cannot identify common types of bias, while analogies are unsuitable indicators. |
A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Word embeddings suffer from unintended demographic biases, a new study shows . word embedders can cause downstream NLP systems to be unfair, the authors argue . |
| Approach: | They propose a metric to evaluate the fairness of word embeddings via the relative negative sentiment associated with demographic identity terms from various protected groups. |
| Outcome: | The proposed metric measures fairness in word embeddings via the relative negative sentiment associated with demographic identity terms from various protected groups. |
From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work in this field has looked most commonly into gender bias, racial bias, and religious bias. |
| Approach: | They propose an algorithm that uses a neural network to perform ‘soft debiasing’ and build on the seminal work of (CITATION) and (CitATION). |
| Outcome: | The proposed algorithm outperforms current methods on gender, race, and religion metrics on a wide range of metrics. |
Discovering Biased News Articles Leveraging Multiple Human Annotations (2020.lrec-1)
Copied to clipboard
| Challenge: | Political propaganda and one-sided views can be found in the news and can cause distrust in media. |
| Approach: | They propose to annotate politically biased news articles by an algorithm annotated by domain experts and crowd workers and to compare them to crowd workers. |
| Outcome: | The proposed method compares domain experts to crowd workers and shows that bias can be detected automatically. |
On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach (2023.acl-short)
Copied to clipboard
| Challenge: | Word embeddings have been used to quantify biases in texts for years, but their statistical properties and advantages have not been studied. |
| Approach: | They propose to use PMI-based metric to quantify bias in corpora by conditional probabilities and odds ratio to approximate it. |
| Outcome: | The proposed measure can be approximated by an odds ratio, which makes statistical inferences cost-effective and meaningful. |
Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing methods to debias word embeddings in binary settings such as gender and religion are limited to binary labels, whereas word2vec embedders can be used to propagate biases. |
| Approach: | They propose a method to debias word embeddings in multiclass settings such as gender and religion, extending the work of Bolukbasi et al. (2016). |
| Outcome: | The proposed method maintains the efficacy in standard NLP tasks while maintaining the utility of embeddings. |