Papers by Karen Zhou
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge (2026.acl-demo)
Copied to clipboard
| Challenge: | AutoChecklist is an open-source library that unifies checklist-based evaluation into composable pipelines. |
| Approach: | They propose an open-source library that unifies checklist-based evaluation into composable pipelines. |
| Outcome: | The open-source library unifies checklist-based evaluation into composable pipelines. |
PRiSM: Benchmarking Phone Realization in Speech Models (2026.acl-long)
Copied to clipboard
Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim, Kwanghee Choi, Eunjung Yeo, Ryan Soh-Eun Shim, Hanyu Zhou, Brendon Boldt, Karen Rosero, Kalvin Chang, Darsh Agrawal, Keer Xu, Chao-Han Huck Yang, Jian Zhu, Shinji Watanabe, David R. Mortensen
| Challenge: | Existing evaluations of phone recognition systems only measure surface-level transcription accuracy. |
| Approach: | They propose to standardize transcription-based evaluation and assess downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. |
| Outcome: | The proposed system outperforms LALMs in clinical, educational, and multilingual settings. |
Entity-Based Evaluation of Political Bias in Automatic Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that NLP systems may encode social biases, but the *political* bias of summarization models remains relatively unknown. |
| Approach: | They use an entity replacement method to examine the portrayal of politicians in automatically generated summaries. |
| Outcome: | The proposed model can control for the content of the source document and can be used to predict the ideal quality of summarization models. |
From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notes (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing automated metrics fail to align with real-world physician preferences. |
| Approach: | They propose a pipeline that distills real user feedback into structured checklists for note evaluation that are interpretable, grounded in human feedback, and enforceable by LLM-based evaluators. |
| Outcome: | The proposed checklist outperforms baseline evaluations in coverage, diversity, and predictive power for human ratings. |