Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on sociodemographic prompting have not explored the effectiveness of this technique. |
| Approach: | They propose to use sociodemographic prompting to steer models towards answers that humans with specific sociodemography would give. |
| Outcome: | The proposed technique can improve zero-shot learning by focusing on human sociodemographic profiles. |
Similar Papers
Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs (2025.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely used to simulate human responses, but their ability to account for demographic differences in subjective tasks remains uncertain. |
| Approach: | They evaluate large language models' ability to understand demographic differences in two subjective judgment tasks: politeness and offensiveness. |
| Outcome: | The proposed models perform better in politeness and offensiveness tasks, while sociodemographic prompting does not improve and worsens their ability to perceive language from sub-populations. |
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemography knowledge. |
| Approach: | They propose to train large language models to be accurate sociodemographic models of annotator variation by using a curated dataset of five tasks with standardized sociodemography. |
| Outcome: | The proposed models improve in sociodemographic prompting when trained but this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemography. |
The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | persona prompting is increasingly used in large language models to simulate views of various sociodemographic groups. |
| Approach: | They use open-source LLMs to study how persona prompts influence LLM simulations . they use role adoption formats and demographic priming strategies to study marginalized groups . |
| Outcome: | The results show that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. |
One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has used personas to study biases by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions. |
| Approach: | They compare six commonly used personacues across seven open and proprietary LLMs on four writing and advice tasks. |
| Outcome: | The proposed model is based on a persona, a synthetic user profile defined by specific attributes, defined by gender or race. |
Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting (2024.emnlp-main)
Copied to clipboard
| Challenge: | Socio-demographic prompting is a commonly employed approach to study cultural biases in LLMs as well as for aligning models to certain cultures. |
| Approach: | They propose to use socio-demographic prompting to probe four LLMs with culturally sensitive and non-sensitive cues on datasets that are supposed to be culturally neutral or sensitive. |
| Outcome: | The proposed model shows significant differences in responses on both kinds of datasets, casting doubt on its robustness. |
Quantifying the Persona Effect in LLM Simulations (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. |
| Approach: | They investigate how integrating persona variables—demographic, social, and behavioral factors—impacts LLMs’ ability to simulate diverse perspectives. |
| Outcome: | The proposed model improves on a zero-shot model with persona prompting. |
Persona Prompting as a Lens on LLM Social Reasoning (2026.eacl-long)
Copied to clipboard
Jing Yang, Moritz Hechtbauer, Elisabeth Khalilov, Evelyn Luise Brinkmann, Vera Schmitt, Nils Feldhus
| Challenge: | Persona prompting (PP) is increasingly used to steer large language models towards user-specific generation, but its effect on rationales remains underexplored. |
| Approach: | They examine how LLM-generated rationales vary when conditioned on different demographic personas . they use word-level rationale annotations to measure agreement with human annotations based on PP . |
| Outcome: | The proposed model improves classification on the most subjective task, but fails to align with real-world demographic counterparts. |
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics (2023.acl-short)
Copied to clipboard
| Challenge: | Existing work has attempted to model individual annotation behaviour rather than predicting aggregated labels. |
| Approach: | They propose to model individual annotator behaviour rather than predicting aggregated labels by adding group-specific layers to multi-annotator models to account for sociodemographics. |
| Outcome: | The proposed model does not significantly improve on toxic content detection tasks. |
Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have tried to evaluate and mitigate social biases accurately using limited prompts. |
| Approach: | They investigate the sensitivity of Large Language Models when changing prompt variations . they found that LLM rankings fluctuate across prompts for both task performance and social bias . |
| Outcome: | The results show that LLM rankings fluctuate when changing prompt variations . |
Which Demographics do LLMs Default to During Annotation? (2025.acl-long)
Copied to clipboard
Johannes Schäfer, Aidan Combs, Christopher Bagdon, Jiahui Li, Nadine Probol, Lynn Greschner, Sean Papay, Yarik Menchaca Resendiz, Aswathy Velutharambath, Amelie Wuehrl, Sabine Weber, Roman Klinger
| Challenge: | Demographics and cultural background of annotators influence the labels they assign in text annotation. |
| Approach: | They examine the attributes of human annotators LLMs inherently mimic and compare them to demographic-conditioned prompts and placebo-conditioned ones. |
| Outcome: | The proposed model incorporates demographics and cultural background into the output of the large language models (LLMs) to evaluate which attributes of human annotators LLMs inherently mimic. |