Papers by Mark Diaz
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives (2024.naacl-long)
Copied to clipboard
Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, Gregory Serapio-García
| Challenge: | Recent work shows that ignoring rater subjectivity is problematic within specific tasks and for specific subgroups. |
| Approach: | They propose a disagreement analysis framework to measure group association in perspectives among different rater subgroups. |
| Outcome: | The proposed framework reveals specific rater groups that have significantly different perspectives than others on certain tasks and helps identify demographic axes that are crucial to consider in specific task contexts. |
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on annotator subjectivity focus on Western contexts and only document differences across age, gender, or racial groups. |
| Approach: | They propose a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K English sentences annotated by a pool of more than 4k annotators from 21 countries. |
| Outcome: | The proposed dataset captures annotators’ moral values along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. |
STAR: SocioTechnical Approach to Red Teaming Language Models (2024.emnlp-main)
Copied to clipboard
Laura Weidinger, John Mellor, Bernat Pegueroles, Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, A. Bergman, Mikel Rodriguez, Verena Rieser, William Isaac
| Challenge: | STAR is a sociotechnical framework that improves on current best practices for red teaming safety of large language models. |
| Approach: | They propose a sociotechnical framework that improves on current best practices for red teaming safety of large language models. |
| Outcome: | The proposed framework improves on current best practices for red teaming safety of large language models. |