Papers by Carolyn Rose
DocLens: Multi-aspect Fine-grained Medical Text Evaluation (2024.acl-long)
Copied to clipboard
Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, Carolyn Rose
| Challenge: | Medical text generation systems are widely used to assist with administrative work and highlight salient information to support decision-making. |
| Approach: | They propose a set of metrics to evaluate completeness, conciseness, and attribution of medical text at a fine-grained level. |
| Outcome: | The proposed framework exhibits substantially higher agreement with medical experts than existing metrics. |
Estimating Agreement by Chance for Sequence Annotation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on chance correction for sequence annotation tasks lack a chance corrected agreement metric. |
| Approach: | They propose a model for generating random annotations which serves as the foundation for estimating chance agreement in sequence annotation tasks. |
| Outcome: | The proposed model is validated in simulation and corpus-based evaluation. |
Programming by Example meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction (2025.acl-long)
Copied to clipboard
Atharva Naik, Darsh Agrawal, Hong Sng, Clayton Marr, Kexun Zhang, Nathaniel Romney Robinson, Kalvin Chang, Rebecca Byrnes, Aravind Mysore, Carolyn Rose, David R. Mortensen
| Challenge: | Historical linguists have written programs that convert reconstructed words into their attested descendants via ordered string rewrite functions. |
| Approach: | They propose to use a model to generate a "similar distribution" for sound law induction . they propose four kinds of methods with varying amounts of inductive bias to investigate best performance . |
| Outcome: | The proposed model shows that it can be fine tuned with training data and evaluation data. |
Deep Neural Model Inspection and Comparison via Functional Neuron Pathways (P19-1)
Copied to clipboard
| Challenge: | a general method for the interpretation and comparison of neural models is proposed . we factor a complex neural model into its functional components . |
| Approach: | They propose a method that factored a complex neural model into its functional components . they use correlated task level and linguistic heuristics to identify correlated pathways . |
| Outcome: | The proposed method can be applied in a purely post-processing manner to understand neural models. |
Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences (2025.findings-emnlp)
Copied to clipboard
Mingqian Zheng, Wenjia Hu, Patrick Zhao, Motahhare Eslami, Jena D. Hwang, Faeze Brahman, Carolyn Rose, Maarten Sap
| Challenge: | Current LLMs are trained to refuse potentially harmful input queries regardless of intent . a study of 480 participants evaluating 3,840 query-response pairs reveals that response strategy largely shapes user experience . |
| Approach: | They examine how different refusal strategies affect user perceptions across varying motivations . they find partial compliance reduces negative user perception by over 50% to flat-out refusals a 480 participants study . |
| Outcome: | The study examines the perceptions of LLMs on user intents and their response strategies . it shows that partial compliance reduces negative user perceptions by over 50% to flat refusals . |
ResPer: Computationally Modelling Resisting Strategies in Persuasive Conversations (2021.eacl-main)
Copied to clipboard
Ritam Dutt, Sayan Sinha, Rishabh Joshi, Surya Shekhar Chakraborty, Meredith Riggs, Xinru Yan, Haogang Bao, Carolyn Rose
| Challenge: | Existing research has failed to account for resisting strategies employed to foil persuasion attempts. |
| Approach: | They propose a framework for identifying resisting strategies in persuasive conversations . they instantiate a dataset comprising persuasion and negotiation conversations based on a hierarchical sequence-labelling neural architecture . |
| Outcome: | The proposed framework is based on two persuasive conversation datasets and leverages a hierarchical sequence-labelling neural architecture to infer resisting strategies automatically. |
Exploring Numeracy in Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing word embeddings are inadequate at capturing numerical properties of numbers. |
| Approach: | They propose to use word embeddings to capture numerical properties of numbers . they hope to develop methods which better capture numeric properties . |
| Outcome: | The proposed models lack the ability to capture numeric properties of numbers, the authors show . their findings provide a starting point for the development of better models . |
Improving Model Factuality with Fine-grained Critique-based Evaluator (2025.acl-long)
Copied to clipboard
Yiqing Xie, Wenxuan Zhou, Pradyot Prakash, Di Jin, Yuning Mao, Quintin Fettes, Arya Talebzadeh, Sinong Wang, Han Fang, Carolyn Rose, Daniel Fried, Hejia Zhang
| Challenge: | Factuality evaluation aims to detect factual errors produced by language models and guide the development of more factual models. |
| Approach: | They propose a framework that leverages FenCE to improve the factuality of LM generators by constructing training data. |
| Outcome: | The proposed framework improves the factuality of LM generators by enhancing their training data. |
MedFilter: Improving Extraction of Task-relevant Utterances through Integration of Discourse Structure and Ontological Knowledge (2020.emnlp-main)
Copied to clipboard
| Challenge: | Identifying task-relevant utterances improves performance at downstream medical processing. |
| Approach: | They propose a novel approach that uses task-oriented conversations to improve utterance classification over SOTA models. |
| Outcome: | The proposed model improves on a corpus of 7,000 doctor-patient conversations on 7,000 patient conversations. |
What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has suggested that political ideology is inherently built into text . a new study examines the impact of experiential factors on annotator perceptions of political ideology . |
| Approach: | They propose to investigate the impact of experiential factors on annotator perceptions of political ideology by analyzing an annotated corpus of political discussion in the U.S. They find that these factors may influence consistency of how political ideologies are perceived by annotators. |
| Outcome: | The findings challenge the assumption that political ideology is built into text . they show that experiential factors may influence how ideologies are perceived . |
Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for language from a social perspective are gaining popularity . we present a generalizable classification approach that leverages Large Language Models . |
| Approach: | They propose a generalizable classification approach that leverages Large Language Models to detect social meaning in conversations. |
| Outcome: | The proposed approach improves on two social meaning detection tasks over 2,340 settings. |
An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated impressive capabilities across complex reasoning and generation tasks. |
| Approach: | They evaluate a broad spectrum of collaboration strategies for repository-level code generation where the weak model handles simpler tasks at lower cost and the most challenging tasks are delegated to the strong model. |
| Outcome: | The proposed model achieves equivalent performance to the strong model while reducing the cost by 40%. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics (2026.findings-acl)
Copied to clipboard
Atharva Naik, null Prakam, Yash Mathur, Darsh Agrawal, Manav Nitin Kapadnis, Yuwei An, Clayton Marr, Carolyn Rose, David R. Mortensen
| Challenge: | a benchmark for inductive reasoning is based on sound law induction in historical linguistics . solve rates are below 5% on hard PBEBench instances with long program cascades despite expensive scaling strategies . |
| Approach: | They propose a benchmark for inductive reasoning inspired by sound law induction in historical linguistics. |
| Outcome: | The proposed approach generates problems with controllable difficulty and ordering constraints . solve rates remain below 5% on hard PBEBench instances with long program cascades . |
AliGATr: Graph-based layout generation for form understanding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State of the art forms understanding models often rely on poorly calibrated output probabilities and low performance on relation extraction tasks. |
| Approach: | They propose a graph-based model that uses a generative objective to represent complex grid-like layouts that are often found in forms. |
| Outcome: | The proposed model performs better on the KIE and RE tasks and is more accurate than existing models. |
Where is this coming from? Making groundedness count in the evaluation of Document VQA models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Document Visual Question Answering (VQA) models have come close to or matching human performance on some benchmarks. |
| Approach: | They propose a method that accounts for the semantic and multimodal groundedness of a model’s outputs and can be parameterized so that users can configure the score according to their preferences. |
| Outcome: | The proposed method produces scores that are a better indicator of a model’s robustness and tends to give higher rewards to better-calibrated answers. |
Keeping Up Appearances: Computational Modeling of Face Acts in Persuasion Oriented Discussions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Politeness principles play a central role in shaping human interaction. |
| Approach: | They propose a generalized framework for modeling face acts in persuasion conversations using an annotated corpus and computational models. |
| Outcome: | The proposed framework reveals differences in face act utilization between asymmetric roles in persuasion conversations and predicts key conversational outcome. |
PerKGQA: Question Answering over Personalized Knowledge Graphs (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for question answering over knowledge graphs have focused on generalizable or generic knowledge, which assumes there is a predefined global KG for all queries. |
| Approach: | They propose to use a non-parametric technique that employs case-based reasoning and a parametric approach using graph neural networks to query a predefined knowledge graph (KG) |
| Outcome: | The proposed methods outperform strong baselines on an academic and an internal dataset by 6.5% and 10.5%. |
Towards Open Domain Event Trigger Identification using Adversarial Domain Adaptation (2020.acl-main)
Copied to clipboard
| Challenge: | supervised event trigger identification models can generalize better across domains . prior work focused on annotating specific categories of events or narratives from specific domains. |
| Approach: | They propose to use adversarial domain adaptation framework to build supervised event trigger identification models which can generalize better across domains. |
| Outcome: | The proposed model improves on literature and news domains with no labeled data. |
Stress Test Evaluation for Natural Language Inference (C18-1)
Copied to clipboard
| Challenge: | Existing models perform well at standard datasets for NLI, achieving impressive results across different genres of text. |
| Approach: | They propose to use automatic stress tests to evaluate models' ability to make inferential decisions. |
| Outcome: | The proposed model performs well across genres of text, but lacks the ability to make inferential decisions. |
Data Augmentation for Code Translation with Comparable Corpora and Multiple References (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for translating code between programming languages are limited by parallel training data. |
| Approach: | They propose a data augmentation technique that builds comparable corpora and augments existing parallel data with multiple reference translations. |
| Outcome: | The proposed techniques improve CodeT5 translation between Java, Python, and C++ by an average of 7.5% Computational Accuracy (CA@1) . |
QA4IE: A Quality Assurance Tool for Information Extraction (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for data annotation do not provide comprehensive support for quality assurance. |
| Approach: | They propose a QA tool for information extraction that detects potential problems in text annotations in a timely manner and accurately assesses the quality of annotations. |
| Outcome: | The proposed tool can detect potential problems in text annotations in a timely manner, accurately assess the quality of annotations, and visually display and summarize annotation discrepancies among annotation team members. |
SOCIAL SCAFFOLDS: A Generalization Framework for Social Understanding Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing tools for social reasoning tasks lack explicit social cues, such as intents and implications. |
| Approach: | They propose a framework for facilitating generalization across social reasoning tasks by generating rationales that make social cues explicit. |
| Outcome: | The proposed framework improves cross-task performance and inferred speaker intent in 44% of cases. |
Towards a new research agenda for multimodal enterprise document understanding: What are we missing? (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we discuss the limitations of multimodal document understanding models in enterprise settings. |
| Approach: | They propose a research agenda that is aimed at driving the field towards higher impact in enterprise applications. |
| Outcome: | The proposed research agenda is aimed at driving the field towards higher impact in enterprise applications. |
Incorporating Multimodal Information in Open-Domain Web Keyphrase Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain Keyphrase extraction (KPE) is a fundamental yet complex NLP task . effective designs encode within layout and formatting signals that point to where the important information can be found. |
| Approach: | They propose a multi-modal approach to open-domain keyphrase extraction (KPE) on the Web that leverages layout and formatting signals to aid in the task. |
| Outcome: | The proposed model outperforms state-of-the-art models on the open-domain keyphrase extraction task. |
Adapting Event Extractors to Medical Data: Bridging the Covariate Shift (2021.eacl-main)
Copied to clipboard
| Challenge: | a new study examines the performance of event extractors to new domains without labeled data . event extraction is a key sub-task of interest for text understanding pipelines in multiple domains . |
| Approach: | They propose to align marginal distributions of source and target domains to adapt event extractors to new domains . they use clinical notes and doctor-patient conversations as a testbed . |
| Outcome: | The proposed models reach F1 scores of 70.0 and 72.9 on notes and conversations respectively. |