FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | evaluating the usefulness of language models for literary-domain tasks remains challenging due to the cost of fine-grained annotation for long-form texts and data contamination concerns inherent in using public-domain literature. |
| Approach: | They use a dataset of long-form, recently written fiction to evaluate embedding models . they prioritize author agency and rely on continual, informed author consent . |
| Outcome: | The proposed dataset of long-form, recently written fiction is compared with existing models on this task. |
Similar Papers
Story Embeddings — Narrative-Focused Representations of Fictional Stories (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to model fictional narratives have focused on the aspect of "what" rather than "how" they are being told. |
| Approach: | They propose a model that embeds stories such that similar stories will result in similar embeddings. |
| Outcome: | The proposed model shows state-of-the-art performance on multiple retrieval tasks and a narrative understanding task. |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |
A Matter of Perspective: Building a Multi-Perspective Annotated Dataset for the Study of Literary Quality (2024.lrec-main)
Copied to clipboard
Yuri Bizzoni, Pascale Feldkamp Moreira, Ida Marie S. Lassen, Mads Rosendahl Thomsen, Kristoffer Nielbo
| Challenge: | a dataset collecting quality judgments on 9,000 English-language novels is presented . authors include experts opinions and crowd-sourced annotations . |
| Approach: | They propose a dataset collecting quality judgments on 9,000 English-language novels by 3,150 predominantly Anglophone authors. |
| Outcome: | The proposed dataset examines the perceived quality of 9,000 English-language novels by 3,150 predominantly anglophone authors. |
Towards A “Novel” Benchmark: Evaluating Literary Fiction with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) context windows have enabled them to process inputs over 100K tokens and generate outputs of up to 10K token. |
| Approach: | They propose a multi-level evaluation framework that incorporates ten metrics across the Macro, Meso, and Micro levels and an annotated fiction dataset. |
| Outcome: | The proposed framework incorporates ten metrics across the Macro, Meso, and Micro levels and is based on a human-human-AI dataset. |
A Survey on LLMs for Story Generation (2025.findings-emnlp)
Copied to clipboard
Maria Teleki, Vedangi Bengali, Xiangjue Dong, Sai Tejas Janjur, Haoran Liu, Tian Liu, Cong Wang, Ting Liu, Yin Zhang, Frank Shipman, James Caverlee
| Challenge: | Methods for story generation with Large Language Models (LLMs) have come into the spotlight recently. |
| Approach: | They propose a novel taxonomy of LLMs for story generation consisting of two major paradigms: independent story generation by an LLM, and author-assistance for story creation . |
| Outcome: | The proposed taxonomy compares existing work on the topic with those of novel author-assistance models. |
An annotated dataset of literary entities (N19-1)
Copied to clipboard
| Challenge: | Existing datasets built on news focus on non-named entities, but not literary texts. |
| Approach: | They propose to annotate 210,532 tokens from 100 different English-language literary texts for ACE entity categories (person, location, geo-political entity, facility, organization, and vehicle). |
| Outcome: | The proposed dataset includes 210,532 tokens drawn from 100 different English-language literary texts. |
LFED: A Literary Fiction Evaluation Dataset for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | LFED is a literary fiction evaluation dataset for large language models that evaluate the capability of LLMs on the long fiction comprehension and reasoning. |
| Approach: | They propose a Literary Fiction Evaluation Dataset to evaluate LLMs' comprehension and reasoning on long fictions. |
| Outcome: | The proposed dataset evaluates the capability of large language models on the long fiction comprehension and reasoning. |
BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies and contain strong layout and stylistic biases. |
| Approach: | They propose a dataset for long-form narrative summarization that uses human written summaries on three levels of difficulty. |
| Outcome: | The proposed dataset covers documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of difficulty. |
Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to document-to-document similarity ranking are limited to relatively short documents or lack similarity labels. |
| Approach: | They propose a self-supervised method for document similarity ranking that can be applied to documents of arbitrary length. |
| Outcome: | The proposed model outperforms existing methods on large documents datasets. |