Papers by Lucy Wang
APPLS: Evaluating Evaluation Metrics for Plain Language Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for plain language summarization (PLS) lack a dedicated assessment metric and the suitability of text generation evaluation metrics is unclear due to unique transformations. |
| Approach: | They propose a granular meta-evaluation testbed to evaluate PLS metrics . they identify four PLS criteria and define perturbations that sensitive metrics should be able to detect . |
| Outcome: | The proposed testbed assesses performance of 14 existing metrics including scores, features, and prompt-based evaluations. |
CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support (2024.findings-acl)
Copied to clipboard
Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Wang, Aakanksha Naik
| Challenge: | Literature review requires researchers to synthesize a large amount of information. |
| Approach: | They propose to use LLMs to generate hierarchical organizations from a set of studies . they use a human-in-the-loop process to correct errors in LLM-generated hierarchies . |
| Outcome: | The proposed model improves assignment of studies to categories by 12.6 F1 points. |
PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning (2026.acl-long)
Copied to clipboard
Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
| Challenge: | Frontier models often lack a view of performance on open-ended, economically consequential tasks in high-stakes professional domains where practical returns matter most. |
| Approach: | They introduce a professional reasoning benchmark that recruits 182 qualified professionals to contribute questions inspired by their workflows. |
| Outcome: | The proposed model outperforms other models in 114 countries and 47 US jurisdictions on hard subsets. |
Personalized Jargon Identification for Enhanced Interdisciplinary Communication (2024.naacl-long)
Copied to clipboard
| Challenge: | Identifying and translating scientific jargon for individual researchers could speed up research, but current methods of jaron identification rely on corpus-level familiarity indicators rather than modeling researcher-specific needs. |
| Approach: | They collect over 10K term familiarity annotations from 11 computer science researchers and investigate supervised and prompt-based methods to predict individual jargon familiarity. |
| Outcome: | The proposed method improves jargon familiarity prediction by using domain, subdomain, and individual knowledge. |
MathFish: Evaluating Language Model Math Reasoning via Grounding in Educational Curricula (2024.findings-emnlp)
Copied to clipboard
| Challenge: | pedagogical experts spend months reviewing published math problems to ensure that they align with critical skills or concepts. |
| Approach: | They propose a novel approach for evaluating language models' mathematical abilities by combining a dataset of 385 fine-grained descriptions of K-12 math skills and concepts with 9.9K math problems labeled with these standards. |
| Outcome: | The proposed model can discern skills and concepts enabled by math content, and it can be used to assess language models' mathematical abilities. |
Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-document summarization (MDS) assumes a set of topic-related documents is provided as input. |
| Approach: | They formalize the task and bootstrap it using existing datasets, retrievers and summarizers. |
| Outcome: | The proposed method reduces the sensitivity of summarizers to imperfect retrieval, but is highly sensitive to other errors. |
TOPICAL: TOPIC Pages AutomagicaLly (2024.naacl-demo)
Copied to clipboard
| Challenge: | Topic pages aggregate useful information about an entity or concept into a single concise article. |
| Approach: | They propose a web app that generates topic pages for biomedical entities on demand . they use large language models and retrieval-augmented generation to generate high-quality topics . |
| Outcome: | The proposed method is based on a human evaluation of 150 biomedical topics . it uses large language models and retrieval-augmented generation (RAG) |
Characterizing LLM Abstention Behavior in Science QA with Context Perturbations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has investigated the ability of LLMs to abstain from answering context-dependent questions when provided insufficient or inconsistent context is provided. |
| Approach: | They propose to improve abstention when provided insufficient or incorrect context . they probed the ability of LLMs to abstain from answering context-dependent science questions . |
| Outcome: | The proposed models abstain from answering science questions when provided insufficient or incorrect context. |