Papers by Hari Shrawgi
SAGE: A Generic Framework for LLM Safety Evaluation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Current safety evaluation methodologies focus on single-turn interactions with generic policies, failing to capture conversational dynamics of real-world usage and application-specific harms. |
| Approach: | They propose a framework for customized and dynamic harm evaluations that employs prompted adversarial agents with diverse personalities based on the Big Five model. |
| Outcome: | The proposed framework enables system-aware multi-turn conversations that adapt to target applications and harm policies. |
Navigating the Cultural Kaleidoscope: A Hitchhiker’s Guide to Sensitivity in Large Language Models (2025.naacl-long)
Copied to clipboard
Somnath Banerjee, Sayan Layek, Hari Shrawgi, Rajarshi Mandal, Avik Halder, Shanu Kumar, Sagnik Basu, Parag Agrawal, Rima Hazra, Animesh Mukherjee
| Challenge: | Cultural harm arises when LLMs misrepresent or normalize values, identities, and practices in ways that conflict with the norms of diverse cultural groups. |
| Approach: | They propose a cultural harm test dataset and a preference dataset to assess model outputs across different cultural contexts. |
| Outcome: | The proposed model improves model behavior significantly reducing the likelihood of generating culturally insensitive or harmful content. |
Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have unlocked unprecedented applications of AI. |
| Approach: | They propose to use a social benchmark to evaluate the bias protection provided by Large Language Models (LLMs) with a variety of tasks with varying complexities to assess their effectiveness. |
| Outcome: | The proposed benchmark shows that both ChatGPT and GPT-4 have strong biases with respect to nationality, gender, race, and religion. |
LLM Safety for Children (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly impacting children through education, toys, and therapy, offering benefits like improved mental health and parental controls. |
| Approach: | They propose a comprehensive approach to evaluating LLM safety specifically for children by listing potential risks that children may encounter when using LLM-powered applications. |
| Outcome: | The proposed model bridges the gap in child safety literature across various fields. |