Do *they* mean ‘us’? Interpreting Referring Expression variation under Intergroup Bias (2024.findings-emnlp)
Copied to clipboard
| Challenge: | We model intergroup bias as a tagging task on English sports comments from forums dedicated to fandom for NFL teams . linguistic descriptions of win probability are used for large-scale analysis of intergroup variation . |
| Approach: | They propose to model intergroup bias as a tagging task on NFL fan comments . they use linguistic models to model the bias and use them to generate large-scale annotations . |
| Outcome: | The proposed model can reveal unobserved variations in the form of referents across win probabilities. |
Similar Papers
Ask LLMs Directly, “What shapes your bias?”: Measuring Social Bias in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate social bias in large language models have limitations . et al., 1995: stereotypes shape social perceptions without objective basis . |
| Approach: | They propose a method to intuitively quantify social perceptions and suggest metrics to evaluate biases within LLMs. |
| Outcome: | The proposed metrics capture the multi-dimensional aspects of social bias, the paper shows . they show that the proposed metrics can be used to evaluate bias in large language models . |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Language Models Predict Empathy Gaps Between Social In-groups and Out-groups (2025.naacl-long)
Copied to clipboard
| Challenge: | Studies of human psychology have shown that people are more motivated to extend empathy to in-group members than out-group member. |
| Approach: | They propose to use language models to study intergroup empathy gap . they use a short description of an experience to predict emotion intensity . |
| Outcome: | The proposed model exhibited strongest intergroup bias among those tested. |
French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English (2022.acl-long)
Copied to clipboard
| Challenge: | We introduce 1,679 sentence pairs in French that cover stereotypes in ten types of bias like gender and age. |
| Approach: | They build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages and cultures. |
| Outcome: | The proposed dataset allows for comparability across languages while characterizing biases that are specific to each country and language. |
Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have tried to evaluate and mitigate social biases accurately using limited prompts. |
| Approach: | They investigate the sensitivity of Large Language Models when changing prompt variations . they found that LLM rankings fluctuate across prompts for both task performance and social bias . |
| Outcome: | The results show that LLM rankings fluctuate when changing prompt variations . |
How people talk about each other: Modeling Generalized Intergroup Bias and Emotion (2023.eacl-main)
Copied to clipboard
Venkata Subrahmanyan Govindarajan, Katherine Atwell, Barea Sinno, Malihe Alikhani, David I. Beaver, Junyi Jessy Li
| Challenge: | Current studies of bias in NLP rely on identifying (unwanted or negative) bias towards a specific demographic group, but this is not always practical. |
| Approach: | They extrapolate a notion of bias from social science literature to predict interpersonal group relationship (IGR) using interpersonal emotions as an anchor. |
| Outcome: | The proposed model predicts the interpersonal group relationship (IGR) using interpersonal emotions as an anchor. |
In-Depth Look at Word Filling Societal Bias Measures (2023.eacl-main)
Copied to clipboard
| Challenge: | Language models (LMs) are ubiquitous in current NLP and have brought undeniable performance improvements for many tasks. |
| Approach: | They propose to use word filling prompts to evaluate language models' behavior to find out if they are valid. |
| Outcome: | The proposed measures produce unexpected and illogical results when appropriate control group samples are constructed. |
Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in simulating human behaviour and social intelligence, but they risk perpetuating societal biases, especially when demographic information is involved. |
| Approach: | They propose a framework that measures semantic shifts in responses and an LLM-judged Preference Win Rate to assess how demographic prompts affect response quality across power-disparate social scenarios. |
| Outcome: | The proposed framework measures semantic shifts in responses and an LLM-judged Preference Win Rate (WR) to assess how demographic prompts affect response quality across power-disparate social scenarios. |
Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas (2024.findings-emnlp)
Copied to clipboard
Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, Brenda Curtis
| Challenge: | Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. |
| Approach: | They propose to prompt LLMs with human-like personas and ask them to answer as if they were a specific human, either explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. |
| Outcome: | The proposed model is based on explicit, explicit, and implicit personas, and fails to show implicit biases. |
Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Representative bias is a tendency of Large Language Models to generate outputs that mirror the experiences of certain identity groups, and affinity bias is an evaluative preference for specific narratives. |
| Approach: | They propose two new metrics to measure representative bias and affinity bias within large language models and present a new set of tasks designed with customized rubrics to detect these biases. |
| Outcome: | The proposed model identifies representative biases in prominent LLMs, with a preference for identities associated with being white, straight, and men. |