Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives? (2026.eacl-long)
Copied to clipboard
Karin De Langis, Püren Öncel, Ryan Peters, Andrew Elfenbein, Laura Kristen Allen, Andreas Schramm, Dongyeop Kang
| Challenge: | Contemporary models of (human) reading comprehension characterize comprehension as a dynamic process in which the reader continually builds and updates representations to maintain coherence and integrate new information with prior knowledge. |
| Approach: | They use a paired narrative dataset to examine the extent to which large language models can reliably separate incoherent and coherent stories. |
| Outcome: | The proposed models do not eliminate the deficits in the model internal state and behavior. |
Similar Papers
Are Large Language Model Temporally Grounded? (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent large language models lack a consistent temporal model of textual narratives . sentence ordering in unlabelled texts is only weakly correlated with event ordering . |
| Approach: | They evaluate LLMs with textual narratives and evaluate their common-sense knowledge . they find that LLM models struggle the most with self-consistency . |
| Outcome: | The proposed models lack a consistent temporal model of textual narratives. |
Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work on coherence assessment using NLMs focuses on properties acquired from stand-alone sentences, but their ability to model discourse and pragmatic phenomena is still unclear. |
| Approach: | They propose to use a Neural Language Model to assess coherence in multiple languages to compare models' performance and to examine their performance in a cross-language scenario. |
| Outcome: | The proposed model can model coherent and incoherent text in multiple languages and in-domain settings. |
How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these models reflect human-like cognition versus advanced pattern recognition remains an open question. |
| Approach: | They conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. |
| Outcome: | The proposed framework can be used to assess the cognitive and linguistic capabilities of large language models (LLMs). |
Do language models have coherent mental models of everyday things? (2023.acl-long)
Copied to clipboard
| Challenge: | Psychologists and cognitive scientists hypothesize that humans develop mental models of the world, namely internal, conceptual representations of the environment which we base our decisions and actions on. |
| Approach: | They propose to add a constraint satisfaction layer to the LM's raw predictions to apply commonsense constraints to reduce incoherence. |
| Outcome: | The proposed extension removes inconsistencies and improves accuracy by 16-20%. |
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a common approach to coherence evaluation is shuffling the sentence order of a text, creating incoherent text samples that need to be discriminated from the original. |
| Approach: | They propose an extendable set of test suites addressing different aspects of discourse and dialogue coherence. |
| Outcome: | The proposed evaluation paradigm is suited to evaluate linguistic qualities that contribute to the notion of coherence. |
Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extractive summarization lack coherence, despite improvements . a human-annotated dataset is used to improve coherency of extractive summary . |
| Approach: | They propose to use human-annotated datasets to create coherent extractive summaries . they use supervised fine-tuning and natural language user feedback to enhance coherence . |
| Outcome: | The proposed dataset shows that LLMs can produce coherent summaries with human feedback. |
Unstructured Minds, Predictable Machines: A Comparative Study of Narrative Cohesion in Human and LLM Stream-of-Consciousness Writing (2025.acl-srw)
Copied to clipboard
| Challenge: | Stream-of-consciousness narratives are a challenge for large language models (LLMs) authors examined differences between human and LLM-generated narratives to assess narrative coherence and personality expression. |
| Approach: | They generate SoC narratives by prompting LLMs with the first half of SoC-essays while either providing the models with the personality characteristics (Big Five) or omitting them. |
| Outcome: | The proposed models showed low similarity between LLM-generated continuations and original human texts, as measured by cosine similarity, perplexity, and BLEU scores. |
Lost in Stories: Consistency Bugs in Long Story Generation by LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors unexplored. |
| Approach: | They propose a benchmark to evaluate narrative consistency in long-form story generation. |
| Outcome: | Evaluating LLMs, we find consistency errors are common in factual and temporal dimensions . authors say the findings can inform future efforts to improve consistency in long-form narrative generation. |
Who Holds the Pen? Caricature and Perspective in LLM Retellings of History (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are increasingly used to simulate human perspectives, authors say . authors: asymmetries in tone, stance, and emphasis can quietly, yet systematically, distort how history is told and remembered. |
| Approach: | They analyze LLM-generated responses across 197 historically significant events . they find that LLMs reliably distinguish persona-based responses from neutral baselines . |
| Outcome: | The findings show that LLMs distinguish persona-based responses from neutral baselines and that directly affected personas exhibit higher exaggeration. |
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |