A Matter of Perspective: Building a Multi-Perspective Annotated Dataset for the Study of Literary Quality (2024.lrec-main)
Copied to clipboard
Yuri Bizzoni, Pascale Feldkamp Moreira, Ida Marie S. Lassen, Mads Rosendahl Thomsen, Kristoffer Nielbo
| Challenge: | a dataset collecting quality judgments on 9,000 English-language novels is presented . authors include experts opinions and crowd-sourced annotations . |
| Approach: | They propose a dataset collecting quality judgments on 9,000 English-language novels by 3,150 predominantly Anglophone authors. |
| Outcome: | The proposed dataset examines the perceived quality of 9,000 English-language novels by 3,150 predominantly anglophone authors. |
Similar Papers
An Annotated Dataset of Coreference in English Literature (2020.lrec-1)
Copied to clipboard
| Challenge: | Using OntoNotes, coreference resolution systems are typically evaluated on this data exclusively. |
| Approach: | They present a new dataset of coreference annotations for works of literature in English covering 29,103 mentions in 210,532 tokens from 100 works of fiction published between 1719 and 1922. |
| Outcome: | The proposed dataset covers 29,103 mentions in 210,532 tokens from 100 works of fiction published between 1719 and 1922. |
FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the usefulness of language models for literary-domain tasks remains challenging due to the cost of fine-grained annotation for long-form texts and data contamination concerns inherent in using public-domain literature. |
| Approach: | They use a dataset of long-form, recently written fiction to evaluate embedding models . they prioritize author agency and rely on continual, informed author consent . |
| Outcome: | The proposed dataset of long-form, recently written fiction is compared with existing models on this task. |
Proposal: From One-Fit-All to Perspective Aware Modeling (2025.acl-srw)
Copied to clipboard
| Challenge: | Variation in human annotation and human perspectives has drawn increasing attention in natural language processing research. |
| Approach: | They propose to use annotation formats that better capture granularity and uncertainty of individual judgments and annotation modeling that leverages socio-demographic features to better represent and predict underrepresented or minority perspectives. |
| Outcome: | The proposed tasks aim to advance natural language processing research towards more faithfully reflecting the diversity of human interpretation, enhancing both inclusiveness and fairness in language technologies. |
An annotated dataset of literary entities (N19-1)
Copied to clipboard
| Challenge: | Existing datasets built on news focus on non-named entities, but not literary texts. |
| Approach: | They propose to annotate 210,532 tokens from 100 different English-language literary texts for ACE entity categories (person, location, geo-political entity, facility, organization, and vehicle). |
| Outcome: | The proposed dataset includes 210,532 tokens drawn from 100 different English-language literary texts. |
Let’s discuss! Quality Dimensions and Annotated Datasets for Computational Argument Quality Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Argumentation is a key competence and an important cultural technique in democratic societies. |
| Approach: | They propose to create domain-specific datasets and methods to assess argument quality. |
| Outcome: | The proposed methods address gaps in the literature and aid future research in the domain. |
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2022.tacl-1)
Copied to clipboard
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
| Challenge: | Lower-resource corpora have systematic issues, including mislabeled or nonstandard/ambiguous language codes. |
| Approach: | They manually audit the quality of 205 language-specific corpora released with five major public datasets. |
| Outcome: | The results show that lower-resource corpora have systematic issues even for non-proficient speakers. |
The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies comparing AI-generated and human-authored literary texts have produced conflicting results. |
| Approach: | They hypothesize that differences in reading quality can be explained by genuine differences in how readers interpret and value literature . |
| Outcome: | The authors show that the differences in reading quality are largely explained by differences in how readers interpret and value literature, rather than by an intrinsic quality of the texts evaluated. |
Personality Understanding of Fictional Characters during Book Reading (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to predict characters' personalities have not been studied in the NLP field due to the lack of appropriate datasets mimicking the process of book reading. |
| Approach: | They propose a dataset to predict characters' personalities that uses an exhaustive vocabulary of personality traits as targets. |
| Outcome: | The proposed dataset is efficient and accurate and relies on long-term context to achieve accurate predictions for both machines and humans. |
Literary Event Detection (P19-1)
Copied to clipboard
| Challenge: | a new dataset of literary events is presented to examine the nature of narratives . literature presents a number of challenges for existing systems, including complex narration . |
| Approach: | They propose a dataset of literary events that are depicted as taking place within the imagined space of a novel. |
| Outcome: | The proposed model achieves an F1 score of 73.9 for prestige and popularity . the best performing model achieve a score of 79.9 for prestige compared to the previous model . |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |