Papers by Jay DeYoung
MSˆ2: Multi-Document Summarization of Medical Studies (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for multi-document summarization (MDS) are either in the general domain, such as WikiSum, or very small such as DUC 1 or TAC 2011 . Existing systems for summarizing biomedical literature take 1-2 years to complete . |
| Approach: | They propose to use a multi-document summarization system based on BART to assess the quality of the summarized biomedical literature. |
| Outcome: | The proposed system has high summarization quality, but significant work remains to achieve it. |
Inferring Which Medical Treatments Work from Reports of Clinical Trials (N19-1)
Copied to clipboard
| Challenge: | Ideally, one would consult all available evidence from relevant clinical trials. however, these results are primarily disseminated in natural language scientific articles. |
| Approach: | They propose a task that involves inferring results from a full-text article describing randomized controlled trials with respect to a given intervention, comparator, and outcome of interest. |
| Outcome: | The proposed task consists of 10,000+ prompts coupled with full-text articles describing randomized controlled trials. |
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2020.acl-main)
Copied to clipboard
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, Byron C. Wallace
| Challenge: | State-of-the-art models in NLP are opaque in terms of how they come to make predictions. |
| Approach: | They propose to release a benchmark to measure the quality of rationales extracted by models and how faithful these rationale are to human annotators. |
| Outcome: | The proposed benchmark will enable researchers to compare models and track progress on interpretable models for NLP. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |