Papers by Georgios Chochlakis
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models’ Posteriors (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is the primary method for performing natural language tasks with Large Language Models. |
| Approach: | They examine whether aggregation is a confounding factor in the modeling of subjective tasks . they find it is possible for minority annotators to better align with LLMs . |
| Outcome: | The proposed method is based on aggregation of annotations in a dataset with appropriate priors. |
Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts (2025.emnlp-main)
Copied to clipboard
Georgios Chochlakis, Peter Wu, Tikka Arjun Singh Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan
| Challenge: | Existing approaches to model complex subjective tasks in natural language are limited by significant variation in annotations. |
| Approach: | They propose a simple in-context learning binary filtering baseline that estimates the reasonableness of a document-label pair. |
| Outcome: | The proposed approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. |
The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage (2026.acl-long)
Copied to clipboard
Preni Golazizian, Elnaz Rahmati, Jackson Trager, Zhivar Sourati, Nona Ghazizadeh, Georgios Chochlakis, Jose J. Alcocer, Kerby Bennett, Aarya Vijay Devnani, Parsa Hejabi, Harry G. Muttram, Akshay Kiran Padte, Mehrshad Saadatinia, Chenhao Wu, Alireza Salkhordeh Ziabari, Michael Sierra-Arévalo, Nicholas Weller, Shrikanth Narayanan, Benjamin A.t. Graham, Morteza Dehghani
| Challenge: | a new study examines the perception of police-civilian traffic stops using respect ratings and free-text rationales from multiple perspectives. |
| Approach: | They propose a traffic-stop dataset annotated with respect ratings and rationales from multiple perspectives . they use a criterion-driven preference data construction framework to predict personalized respect ratings . |
| Outcome: | The proposed framework improves rating prediction performance and rationale alignment across all three annotators. |
Large Language Models Do Multi-Label Classification Differently (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. |
| Approach: | They propose to use initial probability distributions to analyze output distributions of LLMs at each label generation step to find out how LLM models perform multi-label classification. |
| Outcome: | The proposed methods improve alignment and predictive performance over existing methods. |