Papers by Lydia Chilton
STORYSUMM: Evaluating Faithfulness in Story Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating abstractive summarization are lacking in faithfulness evaluation. |
| Approach: | They propose a dataset that measures faithfulness of LLM summaries with localized errors and faithfulness labels for evaluation methods. |
| Outcome: | The proposed method does not achieve more than 70% accuracy on this task. |
Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data (2026.findings-acl)
Copied to clipboard
Xuanming Zhang, Shwan Ashrafi, Aziza Mirsaidova, Amir H. Rezaeian, Miguel Ballesteros, Lydia Chilton, Zhou Yu, Dan Roth
| Challenge: | Recent work has explored reasoning efficiency via test-time scaling and early exit strategies. |
| Approach: | They propose an anytime reasoning framework and the Anytime Index to improve model quality . they also propose an inference-time self-improvement method to produce better intermediate solutions . |
| Outcome: | The proposed method improves on NaturalPlan, AIME, and GPQA datasets and improves reasoning quality and efficiency under budget constraints. |
OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation (2026.acl-long)
Copied to clipboard
Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton, Dakuo Wang
| Challenge: | evaluating LLMs' ability to mimic real user behavior remains an open challenge due to the lack of high-quality, publicly available datasets that capture both the observable actions and the internal reasoning of an actual user. |
| Approach: | They propose a dataset of Observation, Persona, Rationale, and Action collected from real human participants during online shopping sessions. |
| Outcome: | The proposed dataset is the first to evaluate how well current LLMs can accurately simulate the next web action of a specific user. |
StoryWars: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Understanding and generating collaborative stories remains an underexplored area due to the lack of open-domain corpora. |
| Approach: | They propose to use a dataset of 40,000 collaborative stories written by 9,400 different authors from an online platform to generate a multi-task benchmark. |
| Outcome: | The proposed model achieves the best performance on fully-supervised, few-shot, and zero-shot scenarios while achieving the best results on the fully-supervised tasks. |
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022.emnlp-main)
Copied to clipboard
Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, William Yang Wang
| Challenge: | Existing models that generate unsafe text are susceptible to the dangers of unsafe text generation and are deemed unsafe. |
| Approach: | They use a dataset to empirically study commonsense physical safety across various models for text generation and reasoning tasks. |
| Outcome: | The proposed model can generate unsafe text and reject it, but the different harms that can occur do not receive equal attention, which may consequently downplay certain harms. |