Papers by Christopher Homan
GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives (2024.naacl-long)
Copied to clipboard
Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, Gregory Serapio-García
| Challenge: | Recent work shows that ignoring rater subjectivity is problematic within specific tasks and for specific subgroups. |
| Approach: | They propose a disagreement analysis framework to measure group association in perspectives among different rater subgroups. |
| Outcome: | The proposed framework reveals specific rater groups that have significantly different perspectives than others on certain tasks and helps identify demographic axes that are crucial to consider in specific task contexts. |
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Annotator disagreements are resolved before learning takes place, but researchers question the performance of a system when annotators disagree. |
| Approach: | They propose a method that uses language features and label distributions to pool similar items into larger labels. |
| Outcome: | The proposed method is based on five publicly available datasets with varying levels of disagreements on social media and in the wild using a dataset from Facebook. |
Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive (2023.emnlp-main)
Copied to clipboard
Tharindu Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri, Christopher Homan, Ashiqur KhudaBukhsh
| Challenge: | a paper examines how machine and human moderators disagree on offensive speech . offensive speech detection is a key component of content moderation . |
| Approach: | They propose a large-scale noise audit and a vicarious offense dataset to investigate disagreement on social web political discourse. |
| Outcome: | The proposed dataset reveals that moderation outcomes vary wildly across different machine moderators. |
Feriji: A French-Zarma Parallel Corpus, Glossary & Translator (2024.acl-srw)
Copied to clipboard
| Challenge: | MT has seen significant advances in recent years, but the representation of African languages in MT systems is underrepresented due to linguistic complexities and limited resources. |
| Approach: | They propose a first robust parallel French-Zarma corpus and a glossary for MT that contains 61,085 sentences in Zarma and 42,789 in French. |
| Outcome: | The proposed model improves the representation of the Zarma language, a dialect of Songhay, spoken by over 5 million people across Niger and neighboring countries. |
Sensing and Learning Human Annotators Engaged in Narrative Sensemaking (N18-4)
Copied to clipboard
| Challenge: | a substantial sector of the gig economy is the use of crowdworkers to annotate data for machine learning and analysis. |
| Approach: | They propose a narrative-sorting annotation task that sorts tweets chronologically by topic, emotional content, and length. |
| Outcome: | The proposed task enables readers to sort sequential, target-topical, and emotionally emotional tweets. |
Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo (2023.findings-acl)
Copied to clipboard
Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, Christopher Homan
| Challenge: | a recent study shows that annotator disagreement is common in supervised learning . a simple neural model that learns to predict annotators' labels is competitive with other models that do not model specific annotations. |
| Approach: | They propose a neural model that learns to predict annotator distributions by aggregating over all annotators. |
| Outcome: | The proposed model outperforms models that do not model specific annotators or do not learn label distribution learning. |
Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance (2023.findings-acl)
Copied to clipboard
| Challenge: | Among the problems with leaderboard culture in NLP has been the widespread lack of confidence estimation in reported results. |
| Approach: | They propose a framework and simulator for estimating p-values for comparisons between the results of two systems using variance found naturally (though rarely reported) in test set items and individual labels on an item (responses). |
| Outcome: | The proposed framework and simulator are used to estimate p-values for comparisons between the results of two systems under the assumption that the null hypothesis is true. |
Rater Cohesion and Quality from a Vicarious Perspective (2024.findings-emnlp)
Copied to clipboard
Deepak Pandita, Tharindu Cyril Weerasooriya, Sujan Dutta, Sarah Luger, Tharindu Ranasinghe, Ashiqur KhudaBukhsh, Marcos Zampieri, Christopher Homan
| Challenge: | Recent work in reinforcement learning with human feedback (RLHF) highlights the gains in model performance from aligning them to human values. |
| Approach: | They propose to use vicarious annotation to break down disagreement by asking raters how they think others would annotate the data. |
| Outcome: | The proposed method breaks down disagreements by asking raters how they think others would annotate the data. |