| Challenge: | a substantial sector of the gig economy is the use of crowdworkers to annotate data for machine learning and analysis. |
| Approach: | They propose a narrative-sorting annotation task that sorts tweets chronologically by topic, emotional content, and length. |
| Outcome: | The proposed task enables readers to sort sequential, target-topical, and emotionally emotional tweets. |
Similar Papers
Can Machines Resonate with Humans? Evaluating the Emotional and Empathic Comprehension of LMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Empathy plays a pivotal role in fostering prosocial behavior, often triggered by the sharing of personal experiences through narratives. |
| Approach: | They propose to use contrastive learning with masked LMs and supervised fine-tuning with large language models to improve empathy understanding in NLP models. |
| Outcome: | The proposed methods show that there is low agreement among annotators and that cultural differences are a factor in their interpretation of empathy. |
HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empathy is a foundational psychological process that drives many prosocial functions. |
| Approach: | They propose a theory-based taxonomy that delineates elements of narrative style that can lead to empathy with the narrator of a story. |
| Outcome: | The proposed taxonomy delineates elements of narrative style that can lead to empathy with the narrator of a story. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
The Empirical Variability of Narrative Perceptions of Social Media Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Identifying stories in social media texts provides a lens through which we can study how individuals and communities process and communicate experiences. |
| Approach: | They construct a taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales. |
| Outcome: | The proposed model shows that crowd workers disagree on categorical labels, free-text storytelling rationales, authorial intent, and more. |
Who Feels What and Why? Annotation of a Literature Corpus with Semantic Roles of Emotions (C18-1)
Copied to clipboard
| Challenge: | Emotion analysis and classification is a challenging task which has been tackled with relatively straight-forward approaches. |
| Approach: | They propose to annotate emotion trigger phrases and entities in the roles of experiencers, targets, and causes of the emotion in literature by Project Gutenberg. |
| Outcome: | The proposed corpus supports qualitative literary studies and digital humanities. |
Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories (L18-1)
Copied to clipboard
| Challenge: | a new dataset is used to classify text into positive, negative, and neutral classes . a large amount of work on automatic detecting emotions from text has focused on classifying text into basic emotion categories . |
| Approach: | They use Twitter as the source of the textual data they annotate to find out which emotions often present together in tweets . |
| Outcome: | The proposed dataset is useful for training and testing supervised machine learning algorithms . it is based on the results of the SemEval-2018 task 1: Affect in Tweets . |
Modeling Empathic Similarity in Personal Narratives (2023.emnlp-main)
Copied to clipboard
| Challenge: | EmpathicStories is a dataset of 1,500 personal stories annotated with empathic similarity features and 2,000 pairs of stories annnotated by empathism. |
| Approach: | They propose a task to identify similarity in personal stories based on empathic resonance . they use a dataset of 1,500 personal stories annotated with empathism features . |
| Outcome: | The proposed model outperforms semantic similarity models on correlation and retrieval metrics. |
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to label aggregation fail to capture subjective annotations and can lead to biases. |
| Approach: | They propose annotator-aware representations for text for subjective classification tasks that involve learning representations of annotators. |
| Outcome: | The proposed model improves on metrics that assess the performance on capturing individual annotators’ perspectives. |
Narrative Embedding: Re-Contextualization Through Attention (2021.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to narrative event representation uses attention to re-contextualize events across the whole story . a recent study shows that attention is used to attach event semantics to tokens . |
| Approach: | They propose an unsupervised approach to narrative event representation using attention to re-contextualize events across the whole story. |
| Outcome: | The proposed approach achieves state of the art performance on multiple choice and story cloze tasks. |
Proposal: From One-Fit-All to Perspective Aware Modeling (2025.acl-srw)
Copied to clipboard
| Challenge: | Variation in human annotation and human perspectives has drawn increasing attention in natural language processing research. |
| Approach: | They propose to use annotation formats that better capture granularity and uncertainty of individual judgments and annotation modeling that leverages socio-demographic features to better represent and predict underrepresented or minority perspectives. |
| Outcome: | The proposed tasks aim to advance natural language processing research towards more faithfully reflecting the diversity of human interpretation, enhancing both inclusiveness and fairness in language technologies. |