Papers by Eve Fleisig
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue . |
| Approach: | They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information. |
| Outcome: | The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research. |
FairPrism: Evaluating Fairness-Related Harms in Text Generation (2023.acl-long)
Copied to clipboard
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
| Challenge: | FairPrism dataset provides a framework for measuring and mitigating fairness-related harms caused by AI text generation systems. |
| Approach: | They propose a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. |
| Outcome: | FairPrism is a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering harms relating to gender and sexuality. |
Incorporating Worker Perspectives into MTurk Annotation Practices for NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to data collection for natural language processing on Amazon Mechanical Turk (MTurk) are susceptible to issues regarding workers’ rights and poor response quality without considering the perspectives of MTurq workers. |
| Approach: | They conducted a critical literature review and a survey of MTurk workers to address open questions regarding fair payment, worker privacy, data quality, and considering worker incentives. |
| Outcome: | The findings suggest that future studies may better account for MTurk workers’ experiences in order to respect workers' rights and improve response quality. |
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
| Approach: | They argue that disparities in scale are transient and researchers can work to reduce them . they argue that data, rather than hardware, is still a bottleneck for many applications . |
| Outcome: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
Ghostbuster: Detecting Text Ghostwritten by Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Ghostbuster is a system that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features. |
| Approach: | They propose a method that passes documents through weaker language models, runs a structured search over possible combinations of their features, and trains a classifier on the selected features. |
| Outcome: | The proposed method outperforms existing detectors and a new baseline on student essays, creative writing, and news articles. |
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have questioned assumptions of majority vote aggregated labels. |
| Approach: | They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. |
| Outcome: | The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators. |
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree (2024.emnlp-main)
Copied to clipboard
| Challenge: | Disagreement among annotators can reveal nuances in subjective tasks that lack a simple ground truth . |
| Approach: | They propose three approaches to predict annotator ratings on the toxicity of text . they integrate annotators' history, demographics, survey information into their models . |
| Outcome: | The proposed approach outperforms other methods in toxicity rating prediction. |
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness (2025.naacl-long)
Copied to clipboard
| Challenge: | As models evolve, datasets can become outdated. |
| Approach: | They propose a human-grounded evaluation metric that assesses adversarialness by capturing models’ and humans’ varying abilities, while also identifying poor examples. |
| Outcome: | The proposed evaluation metric measures the accuracy of an adversarial question answering dataset and determines whether models are performing well on the dataset. |
Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | toxicity detection models focus on marginalized groups, but they obscure harms faced by intersectional subgroups. |
| Approach: | They use outlier detection to identify text about people with demographic attributes distant from the "norm" they find model performance is worse for demographic outliers than non-outliers . |
| Outcome: | The proposed model performance is worse for outliers than non-outliers, the authors say . their analysis also shows that outlier analysis can identify harms faced by intersectional groups . |
Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination (2024.emnlp-main)
Copied to clipboard
| Challenge: | a large-scale study of linguistic bias exhibited by ChatGPT covers 10 dialects of English . standard varieties of English, especially SAE, dominate available training data . |
| Approach: | They use ChatGPT to generate models that default to "standard" varieties of English . they also use a feature annotation and native speaker evaluation to analyze the responses . |
| Outcome: | The proposed models default to "standard" varieties of English, but non-"standard" ones exhibit stereotyping, demeaning content, lack of comprehension, condescending responses. |
AI, Take the Wheel: What Drives Delegation and Trust in Human–Computer Cooperative Question Answering? (2026.findings-acl)
Copied to clipboard
Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Zhu Irene Ying, Tianyi Zhou, Jordan Lee Boyd-Graber
| Challenge: | Human-AI collaboration is already happening, both in proactive delegation and deliberative adoption settings. |
| Approach: | They study delegating a task to AI without seeing its output and evaluating AI suggestions to decide whether to adopt them how AI output shapes final decisions. |
| Outcome: | The proposed game pairs 23 experts with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. |
GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration (2025.acl-long)
Copied to clipboard
| Challenge: | Language models are often miscalibrated, leading to confidently incorrect answers. |
| Approach: | They propose a benchmark for language model calibration that incorporates comparison with human calibration. |
| Outcome: | The proposed metric analyzes model calibration errors and identifies types of miscalibration that differ from human behavior. |