| Challenge: | a new task to evaluate text-to-image generation models for multicultural scenes is unexplored. |
| Approach: | They propose a benchmark task to evaluate text-to-image models in multicultural settings . they use a dataset of 9,000 images spanning five countries, three age groups, two genders, 25 historical landmarks, and five languages to analyze behavior . |
| Outcome: | The proposed benchmark analyzes the behavior of state-of-the-art models across multiple dimensions including alignment, image quality, aesthetics, knowledge, and fairness. |
Similar Papers
Diffusion Models Through a Global Lens: Are They Culturally Inclusive? (2025.acl-long)
Copied to clipboard
Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, Alice Oh
| Challenge: | Text-to-image diffusion models have produced compelling, detailed images from text prompts, but their ability to accurately represent cultural nuances remains an open question. |
| Approach: | They propose a benchmark to evaluate whether diffusion models can generate culturally specific images spanning ten countries. |
| Outcome: | The proposed model fails to generate culturally specific images spanning ten countries . it shows significant disparities in cultural relevance, description fidelity, and realism compared to real-world reference images. |
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing text-to-image models fail to produce appropriate images for cultural concepts or objects not well known or underrepresented in western cultures, such as 'hangari' (a Korean utensil). |
| Approach: | They propose a method which iteratively refines the prompt to improve the alignment between the generated images and underrepresented cultural nouns in text-to-image models. |
| Outcome: | The proposed approach improves the alignment between the generated images and cultural nouns in text-to-image models. |
RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation (2025.findings-naacl)
Copied to clipboard
Viacheslav Vasilev, Julia Agafonova, Nikolai Gerasimenko, Alexander Kapitanov, Polina Mikhailova, Evelina Mironova, Denis Dimitrov
| Challenge: | Text-to-image generation models exhibit a strong bias toward English-speaking cultures, ignoring or misrepresenting the unique characteristics of other language groups, countries, and nationalities. |
| Approach: | They propose a RusCode benchmark to evaluate the quality of text-to-image generation containing elements of the Russian cultural code. |
| Outcome: | The proposed model is based on 1250 text prompts in Russian and their translations into English. |
Multilingual Text-to-Image Generation Magnifies Gender Stereotypes (2025.acl-long)
Copied to clipboard
Felix Friedrich, Katharina Hämmerl, Patrick Schramowski, Manuel Brack, Jindřich Libovický, Alexander Fraser, Kristian Kersting
| Challenge: | Text-to-image (T2I) generation models have great results in image quality, flexibility, and text alignment, but they suffer from substantial gender bias. |
| Approach: | They propose a benchmark to study gender bias in multilingual T2I models . they use multilingual prompts to account for grammatical differences influencing gender . |
| Outcome: | The proposed benchmark shows strong gender biases and language-specific differences across models. |
How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-image generative models can generate high-quality photo-realistic images conditional on natural language text descriptions in a zero-shot fashion. |
| Approach: | They propose an Ethical NaTural Language Interventions in Text-to-Image GENeration benchmark dataset to evaluate the change in image generation conditional on ethical interventions across three social axes – gender, skin color, and culture. |
| Outcome: | The proposed model generations cover diverse social groups while preserving image quality. |
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025.findings-emnlp)
Copied to clipboard
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste, Yash Goyal, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | CulturalFrames is a benchmark designed for rigorous human evaluation of cultural representation in visual generations. |
| Approach: | They propose to quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) and implicit (unstated, implied by the prompt’s cultural context) cultural expectations. |
| Outcome: | The proposed model is based on 983 prompts, 3637 images and 10k human annotations from 10 countries and 5 socio-cultural domains. |
ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation (2024.acl-long)
Copied to clipboard
Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan Reddy, Sunipa Dev
| Challenge: | Existing approaches for evaluating stereotypes have a noticeable lack of coverage of global identity groups and their associated stereotypes. |
| Approach: | They propose to use a dataset to evaluate nationality-based stereotypes in T2I models across 135 nationalities to assess offensive stereotypes. |
| Outcome: | The proposed dataset enables evaluation of known nationality-based stereotypes across 135 nationalities. |
Multilingual Generation in Abstractive Summarization: A Comparative Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for multilingual generation lack thorough analysis due to extensive linguistic diversity. |
| Approach: | They propose to classify multilingual generation methodologies into three categories based on their underlying modeling principles . they introduce an automatic metric to mitigate spurious correlations associated with language mixing . |
| Outcome: | The proposed model improves in high-resource, low-resourced, and zero-shot scenarios. |
Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for text-to-image generation models is proposed to address these limitations . SSA focuses on learning structured semantic embeddings across different modalities . |
| Approach: | They propose a method to evaluate text-to-image generation models using structured semantic embeddings . they propose to learn mutated prompts by substituting words with equivalent or nonequivalent alternatives . |
| Outcome: | The proposed method improves the measurement of semantic consistency of text-to-image generation models. |
From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown emerging capabilities through large-scale training that have made them gain popularity in recent years. |
| Approach: | They propose to perform retrieval across universals and cultural visual grounding tasks to assess cultural diversity across universal and culture-specific local concepts. |
| Outcome: | The proposed benchmarks show that the models perform significantly across cultures, underscoring the need for enhancing multicultural understanding in vision-language models. |