Papers by Melanie Subbiah
Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing fact-checking benchmarks require systems to verify claims from everyday text against evidence from scientific journal articles. |
| Approach: | They propose a benchmark system that checks claims from news against scientific journal articles and veracity labels. |
| Outcome: | The new benchmark achieves F1 scores of 76.99 and 69.90 on both a fact-checking specific system and GPT-3.5, respectively. |
Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | In many domains, determining faithfulness of a claim to a source document is a binary judgment . but, whether a document is factual or whether it is entailed given some input is highly subjective. |
| Approach: | They propose a task to manage the subjectivity involved with factuality judgments of ambiguous claims. |
| Outcome: | The proposed method improves the annotator agreement on faithfulness of a claim by 21%. |
Unsupervised Selective Rationalization with Noise Injection (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised selective rationalization produces rationales alongside predictions, but does not ensure that the rationale contains a plausible explanation for the prediction. |
| Approach: | They propose a technique that injects noise between a rationale generator and a predictor to limit generation of implausible rationales. |
| Outcome: | The proposed method achieves significant improvements in plausibility and task accuracy over the state-of-the-art models while maintaining or improving model faithfulness. |
STORYSUMM: Evaluating Faithfulness in Story Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating abstractive summarization are lacking in faithfulness evaluation. |
| Approach: | They propose a dataset that measures faithfulness of LLM summaries with localized errors and faithfulness labels for evaluation methods. |
| Outcome: | The proposed method does not achieve more than 70% accuracy on this task. |
Mitigating Covertly Unsafe Text within Natural Language Systems (2022.findings-emnlp)
Copied to clipboard
Alex Mei, Anisha Kabir, Sharon Levy, Melanie Subbiah, Emily Allaway, John Judge, Desmond Patton, Bruce Bimber, Kathleen McKeown, William Yang Wang
| Challenge: | Existing studies on text safety have focused on overtly unsafe, covertly, or indirectly unsafe statements. |
| Approach: | They propose a method to identify physical harm-causing statements as overtly, covertly or indirectly unsafe and a solution to mitigate the generation of such statements. |
| Outcome: | The proposed methods identify the type of unsafe language that can cause physical harm and identify mitigation strategies to inspire future researchers to tackle this challenging problem. |
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022.emnlp-main)
Copied to clipboard
Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, William Yang Wang
| Challenge: | Existing models that generate unsafe text are susceptible to the dangers of unsafe text generation and are deemed unsafe. |
| Approach: | They use a dataset to empirically study commonsense physical safety across various models for text generation and reasoning tasks. |
| Outcome: | The proposed model can generate unsafe text and reject it, but the different harms that can occur do not receive equal attention, which may consequently downplay certain harms. |