Papers by Karolina Stanczak
Measuring Intersectional Biases in Historical Documents (2023.findings-acl)
Copied to clipboard
Nadav Borenstein, Karolina Stanczak, Thea Rolskov, Natacha Klein Käfer, Natália da Silva Perez, Isabelle Augenstein
| Challenge: | digitised historical documents suffer from errors introduced by optical character recognition (OCR) and are written in an archaic language. |
| Approach: | They investigate the continuities and transformations of bias in Caribbean historical newspapers during the colonial era . they use distributional semantics models and word embeddings to measure gender, race, and intersectional biases. |
| Outcome: | The authors show that gender and racial biases are interdependent and their intersection triggers distinct effects. |
Benchmarking Vision Language Models for Cultural Understanding (2024.emnlp-main)
Copied to clipboard
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Steenkiste, Lisa Hendricks, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | Recent multimodal vision-language models have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning. |
| Approach: | They propose a visual question-answering benchmark to assess VLMs' cultural understanding of various facets of culture from 11 countries across 5 continents. |
| Outcome: | The visual question-answering benchmark aims to assess VLMs' cultural understanding across regions. |
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025.findings-emnlp)
Copied to clipboard
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste, Yash Goyal, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | CulturalFrames is a benchmark designed for rigorous human evaluation of cultural representation in visual generations. |
| Approach: | They propose to quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) and implicit (unstated, implied by the prompt’s cultural context) cultural expectations. |
| Outcome: | The proposed model is based on 983 prompts, 3637 images and 10k human annotations from 10 countries and 5 socio-cultural domains. |
Probing for Reading Times (2026.acl-long)
Copied to clipboard
Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu, Mario Giulianelli, Karolina Stanczak, Ryan Cotterell
| Challenge: | a large body of work on probing has demonstrated that language model representations encode a wealth of linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. |
| Approach: | They use regularized linear regression to compare language model representations against scalar predictors. |
| Outcome: | The representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. |
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual pre-trained models can learn to generalise across languages . however, it remains unclear how these models learn to learn multilingual representations . |
| Approach: | They propose a hypothesis that multilingual pre-trained models can derive language-universal abstractions about grammar by aligning morphosyntactic markers that fulfil a similar grammatical function across languages. |
| Outcome: | The proposed model can derive language-universal abstractions even without explicit supervision. |
Social Bias Probing: Fairness Benchmarking for Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating social biases in language models have been limited to binary association tests on small datasets. |
| Approach: | They propose a framework for probing language models for social biases by assessing disparate treatment . they use a large-scale benchmark to examine the diversity of identities and stereotypes . |
| Outcome: | The proposed framework expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. |
SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models (2025.naacl-long)
Copied to clipboard
Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, Dragomir Radev, Beatrice Savoldi, Vipul Raheja, Jeremy Qin, Esther Ploeger, Arjun Subramonian, Kaustubh Dhole, Kaiser Sun, Amirbek Djanibekov, Jonibek Mansurov, Kayo Yin, Emilio Villa Cueva, Sagnik Mukherjee, Jerry Huang, Xudong Shen, Jay Gala, Hamdan Al-Ali, null Tair Djanibekov, Nurdaulet Mukhituly, Shangrui Nie, Shanya Sharma, Karolina Stanczak, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Marcelo Viridiano, Oskar Van Der Wal, Adina Yakefu, Aurélie Névéol, Mike Zhang, Sydney Zink, Zeerak Talat
| Challenge: | Large Language Models reproduce and exacerbate social biases present in training data, and resources to quantify this issue are limited. |
| Approach: | They propose a multilingual parallel dataset to examine culturally-specific stereotypes that may be learned by LLMs. |
| Outcome: | The proposed dataset includes stereotypes from 20 regions around the world and 16 languages, spanning multiple identity categories subject to discrimination worldwide. |