Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialogues (2022.lrec-1)
Copied to clipboard
Sandhya Singh, Prapti Roy, Nihar Sahoo, Niteesh Mallela, Himanshu Gupta, Pushpak Bhattacharyya, Milind Savagaonkar, Nidhi Sultan, Roshni Ramnani, Anutosh Maitra, Shubhashis Sengupta
| Challenge: | Movies reflect society and also hold power to transform opinions. |
| Approach: | They propose to annotate movie scripts for identity bias using a dataset that is annotated for gender, race/ethnicity, religion, age, occupation, LGBTQ, and other . |
| Outcome: | The proposed dataset contains dialogue turns annotated for gender, race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains biases like body shaming, personality bias, etc. |
Similar Papers
Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detection of biases in contextual language models are inconsistent and inconclusive. |
| Approach: | They propose to use word embedding association test to detect biases in contextual language models to compare them with other methods. |
| Outcome: | The proposed methods are inconsistent and inconclusive for language models with word embeddings. |
Unveiling Identity Biases in Toxicity Detection : A Game-Focused Dataset and Reactivity Analysis Approach (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing datasets focused on gender or racial biases are not designed for the gaming industry, a concern for models built for toxicity detection in videogames’ written chat. |
| Approach: | They propose to use reactivity analysis to highlight oversensitive terms using a language model developed by Ubisoft for toxicity detection on videogame’s written chat and Perspective API to generate a list of terms that trigger the models to varying degrees. |
| Outcome: | The proposed model can detect and amplify identity biases in annotated language models and is compared with a language model developed by Ubisoft for toxicity detection on videogames’ written chat and Perspective API. |
MovieSum: An Abstractive Summarization Dataset for Movie Screenplays (2024.findings-acl)
Copied to clipboard
| Challenge: | Movie screenplay summarization requires an understanding of long input contexts and elements unique to movies. |
| Approach: | They propose a dataset for movie screenplay summarization that includes movie screenplayers accompanied by their Wikipedia plot summaries. |
| Outcome: | The proposed dataset includes 2200 movie screenplays accompanied by their Wikipedia plot summaries. |
Biasly: An Expert-Annotated Dataset for Subtle Misogyny Detection and Mitigation (2024.findings-acl)
Copied to clipboard
Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Smith, Tamara Kneese, Carolyne Pelletier, Ioana Baldini, Yue Dong
| Challenge: | the Biasly dataset captures misogyny in movies in ways unique within the literature. |
| Approach: | The Biasly dataset captures misogyny in North American film by combining annotations of movie subtitles with common NLP algorithms. |
| Outcome: | The Biasly dataset captures misogyny expressions in North American film . it contains annotations of movie subtitles and text generation for rewrites . |
Bias and Fairness in Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | a tutorial will review the history of bias and fairness studies in machine learning and language processing . |
| Approach: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it presents recent community effort to quantify and mitigat bias in natural language processing models . |
| Outcome: | This tutorial reviews the history of bias and fairness studies in machine learning and language processing . it aims to quantify and mitigate bias in natural language processing models for a wide spectrum of tasks . |
AligNarr: Aligning Narratives on Movies (2021.acl-short)
Copied to clipboard
| Challenge: | Experimental results show the viability of an unsupervised approach to align movie scripts with plot summaries. |
| Approach: | They propose an unsupervised method to align movie scripts with plot summaries using a global optimization model. |
| Outcome: | The proposed method outperforms a baseline alignment model on ten movies with 76% F1 score. |
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found evidence of gender bias in machine translation and coreference resolution models using mostly synthetic diagnostic datasets. |
| Approach: | They propose a semi-automatic method to vastly extend synthetic, small diagnostic datasets to include grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments. |
| Outcome: | The proposed method extends the existing dataset to 108K diverse English sentences. |
From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work in this field has looked most commonly into gender bias, racial bias, and religious bias. |
| Approach: | They propose an algorithm that uses a neural network to perform ‘soft debiasing’ and build on the seminal work of (CITATION) and (CitATION). |
| Outcome: | The proposed algorithm outperforms current methods on gender, race, and religion metrics on a wide range of metrics. |
NLPositionality: Characterizing Design Biases of Datasets and Models (2023.acl-long)
Copied to clipboard
| Challenge: | Design biases in NLP systems often stem from creator’s positionality, i.e., views and lived experiences shaped by identity and background. |
| Approach: | They propose a framework for characterizing design biases and quantifying the positionality of NLP datasets and models. |
| Outcome: | The proposed framework characterizes design biases and quantifies alignment with dataset labels and model predictions. |
Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision–Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in self-supervised training have led to a new class of pretrained vision–language models. |
| Approach: | They propose a visual and textual bias benchmark to assess bias in self-supervised multimodal models using 3,800 images and phrases from 14 population subgroups. |
| Outcome: | The proposed model shows that it favors certain groups while maintaining the accuracy of the model. |