Papers by Fabrice Harel-Canada
Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Data augmentation techniques generate low-quality texts with incorrect labels . a new technique is needed to winnow out texts with inaccurate labels based on provenance inspection . |
| Approach: | They develop a data inspection technique that uses provenance inspection and assistive labeling to winnow out texts with incorrect labels. |
| Outcome: | a new human-in-the-loop data inspection technique can winnow out texts with incorrect labels . the technique can reduce human inspection effort by combining provenance inspection and assistive labeling . |
Measuring Psychological Depth in Language Models (2024.emnlp-main)
Copied to clipboard
Fabrice Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Yildiz, Miryung Kim, Amit Sahai, Nanyun Peng
| Challenge: | Current evaluations of creative stories focus on objective properties of the text, such as its style, coherence, diversity, and creativity. |
| Approach: | They propose a framework that measures an LLM's ability to produce authentic and narratively complex stories that provoke emotion, empathy, and engagement. |
| Outcome: | The proposed framework shows that humans can consistently evaluate stories based on the PDS (0.72 Krippendorff’s alpha). |
EnDex: Evaluation of Dialogue Engagingness at Scale (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that measure engagement use expensive human annotas and abstract definitions of the term. |
| Approach: | They propose a human-reaction based model to evaluate dialogue engagingness . they propose combining distant-supervision with a theoretical foundation for engagement . |
| Outcome: | The proposed model is trained on 80k Reddit-based engagement datasets . it uses distant-supervision from human-reaction feedback to evaluate dialogue engagementness . |
Sibylvariant Transformations for Robust Text Classification (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing text transformation techniques are limited in their ability to expand input space . many techniques can artificially expand labeled training sets or test suites, but are class-preserving . |
| Approach: | They propose a concept of sibylvariance to describe transforms that relax the label-preserving constraint and knowably vary the expected class. |
| Outcome: | The proposed transforms can expand input space, but they are limited in their ability to expand . the proposed transform can knowably vary the expected class and lead to more diverse distributions . |