Papers by Hannah Rashkin
Help Me Write a Story: Evaluating LLMs’ Ability to Generate Writing Feedback (2025.acl-long)
Copied to clipboard
| Challenge: | Current models provide specific and mostly accurate writing feedback, but they fail to identify the biggest writing issue in the story and to correctly decide when to offer critical vs. positive feedback. |
| Approach: | They propose a task that corrupts 1,300 stories to intentionally introduce writing issues to study model performance. |
| Outcome: | The proposed model performs well in a controlled task with human and automatic evaluation metrics. |
Modeling Naive Psychology of Characters in Simple Commonsense Stories (P18-1)
Copied to clipboard
| Challenge: | Understanding a narrative requires reasoning about the causal links between the events in the story and the mental states of the characters, even when those relationships are not explicitly stated. |
| Approach: | They propose a new annotation framework to explain naive psychology of story characters as fully-specified chains of mental states with respect to motivations and emotional reactions. |
| Outcome: | The proposed framework provides a baseline performance on several new tasks suggesting avenues for future research. |
Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue (2024.eacl-long)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogues require a balance between being specific to what the conversation partner has said and being attributable to an underlying source document. |
| Approach: | They propose a framework that allows to experiment with various plan variables supported by prior work . they show that metric-aware planning mechanisms are better at automatic evaluations but underperform in human judgment compared to metric agnostic mechanisms. |
| Outcome: | The proposed framework supports metric-agnostic and metric aware content planning, but it underperforms in human judgment. |
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2022.emnlp-main)
Copied to clipboard
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter, Hannaneh Hajishirzi, Mari Ostendorf, Gaurav Singh Tomar
| Challenge: | Existing models for conversational question answering require specific retrievers to understand user questions. |
| Approach: | They develop a query rewriting model CONQRR that rewrites a conversational question into a standalone question. |
| Outcome: | The proposed model achieves state-of-the-art on an open-domain conversational question answering dataset and is effective for two different off-the shelf retrievers. |
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2022.tacl-1)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogue systems powered by large language models often generate responses that, while fluent, are not attributable to a relevant source of information. |
| Approach: | They propose to evaluate the validity of 12k dialogue turns generated by neural dialogue systems trained on three knowledge-grounded dialogue corpora and to use them to analyze eight evaluation metrics. |
| Outcome: | The proposed evaluation metrics rely on spurious correlations, do not reliably distinguish attributable abstractive responses from unattributable ones, and perform substantially worse when the knowledge source is longer. |
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (P19-1)
Copied to clipboard
| Challenge: | EmpatheticDialogues dataset provides a benchmark for empathetic dialogue generation . human evaluators perceive dialogue models as more epathetic . |
| Approach: | They propose a benchmark for empathetic dialogue generation from a dataset of 25k conversations grounded in emotional situations. |
| Outcome: | The proposed benchmarks show that existing models are perceived to be more empathetic by human evaluators compared to models trained on large-scale Internet conversations. |
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (P19-1)
Copied to clipboard
| Challenge: | Existing studies on commonsense knowledge base construction only store loosely structured open-text descriptions of knowledge. |
| Approach: | They propose a commonsense knowledge base construction model that generates rich commonsensense descriptions in natural language. |
| Outcome: | The proposed models can generate rich and diverse commonsense descriptions in natural language. |
Social IQa: Commonsense Reasoning about Social Interactions (D19-1)
Copied to clipboard
| Challenge: | Social IQa contains 38,000 multiple choice questions for probing emotional and social intelligence in a variety of everyday situations. |
| Approach: | They propose a crowdsourcing framework that collects commonsense questions along with correct and incorrect answers about social interactions. |
| Outcome: | The proposed framework mitigates stylistic artifacts in incorrect answers by asking workers to provide the right answer to a different but related question. |
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Unconscious biases continue to be prevalent in modern text and media, calling for algorithms that can assist writers with bias correction. |
| Approach: | They propose a new revision task that debiases text through the lens of connotation frames to correct implicit biases in character portrayals. |
| Outcome: | The proposed approach outperforms existing methods and ablations in the literature. |
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features (2021.acl-long)
Copied to clipboard
| Challenge: | Existing systems that strive to be informative teachers are difficult to build . knowledge grounded dialogue systems are difficult because of limited training objectives . |
| Approach: | They propose to train a generative neural dialogue model that is controlled to stay faithful to evidence . they propose to use additional inputs to generate more objective responses . |
| Outcome: | The proposed model produces responses that are perceived by humans to be objective and faithful to evidence. |
PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel task is to generate a coherent narrative consistent with an outline . large-scale language models are not sufficient in generating coherent narratives for the given outline despite their impressive generation performance . |
| Approach: | They propose a task of outline-conditioned story generation that generates a coherent narrative . they propose 'plotmachines' that tracks dynamic plot states and learns different writing styles . |
| Outcome: | The proposed model can generate a coherent story by tracking the dynamic plot states while conditioning on the input outline while generating the full story. |
Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Simplifying the entire text makes it understandable but sometimes removes important details. |
| Approach: | They propose a simplification task for rewriting text to help readers comprehend text containing unfamiliar concepts and introduce a dataset of 22k definitions from 13 academic domains paired with a difficult concept within each definition. |
| Outcome: | The proposed model outperforms open-source and commercial models on the task and human judges prefer explanations over simplifications of the difficult concept. |
Event2Mind: Commonsense Inference on Events, Intents, and Reactions (P18-1)
Copied to clipboard
| Challenge: | Using a crowdsourced corpus of 25,000 event phrases, we construct a new task that uses commonsense reasoning to reason about the likely intents and reactions of the event participants. |
| Approach: | They construct a crowdsourced corpus of 25,000 event phrases and use them to construct 'commonsense inference' they demonstrate that neural encoder-decoder models can compose embedding representations of previously unseen events and reason about the likely intents and reactions of the event participants. |
| Outcome: | The proposed task can be used to uncover implicit gender inequality in movie scripts. |