Papers by Bailey Kuehl
CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support (2024.findings-acl)
Copied to clipboard
Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Wang, Aakanksha Naik
| Challenge: | Literature review requires researchers to synthesize a large amount of information. |
| Approach: | They propose to use LLMs to generate hierarchical organizations from a set of studies . they use a human-in-the-loop process to correct errors in LLM-generated hierarchies . |
| Outcome: | The proposed model improves assignment of studies to categories by 12.6 F1 points. |
MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting (2022.naacl-main)
Copied to clipboard
| Challenge: | Citation context analysis (CCA) is an important task in natural language processing that studies how and why scholars discuss each other’s work. |
| Approach: | They propose to use a dataset of 12.6K citation contexts from 1.2K computational linguistics papers to model three important CCA phenomena. |
| Outcome: | The proposed dataset contains 12.6K citation contexts from 1.2K computational linguistics papers and can model these phenomena. |
Generating Scientific Claims for Zero-Shot Scientific Fact Checking (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for scientific fact checking require domain expertise and time consuming. |
| Approach: | They propose a new supervised method for generating claims from scientific sentences and a novel method for negating claims. |
| Outcome: | The proposed method improves on existing methods on biomedical claims and negations. |
PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents (2023.emnlp-demo)
Copied to clipboard
Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Chang, Russell Authur, Erin Bransom, Stefan Candra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, Amanpreet Singh, Chris Wilhelm, Angele Zamarron, Marti A. Hearst, Daniel Weld, Doug Downey, Luca Soldaini
| Challenge: | Existing tools for working with scientific documents are limited and documents are often in difficult-to-use PDF formats. |
| Approach: | They propose an open-source Python toolkit for analyzing and processing visually-rich scientific documents. |
| Outcome: | PaperMage provides turn-key recipes for common scientific document processing use-cases. |
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023.eacl-main)
Copied to clipboard
| Challenge: | Human evaluation is labor-intensive, expensive to scale, and difficult to design. |
| Approach: | They propose a set of guidelines for human evaluation of faithfulness in long-form summaries that address the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can our annotator minimize workload while maintaining accurate faithfulness? |
| Outcome: | The proposed framework reduces inter-annotator variance in faithfulness scores while minimizing annotator workload while maintaining accuracy. |
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews (2024.acl-long)
Copied to clipboard
| Challenge: | Existing systems that can interpret complex writing feedback and edit documents in response are limited on the most demanding writing tasks. |
| Approach: | They propose to use peer feedback to revise scientific papers based on peer feedback . they provide labels linking each reviewer comment to the specific paper edits made by the author . |
| Outcome: | The proposed model fails to identify which edits correspond to a comment and the original paper. |
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (2025.acl-demo)
Copied to clipboard
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
| Challenge: | tracing language models' outputs back to training data is a problem because they are trained on text corpora with trillions of tokens . existing methods for tracers have not been scaled to work within this multi-trillion-token setting . |
| Approach: | They propose a system that traces language models' outputs verbatim back to training data . OLMOTRACE retrieves documents from the model's training data that contain exact matches . |
| Outcome: | The proposed system can find verbatim matches between LM output and training data . it can be used to explore fact checking, hallucination, and creativity of language models . |
CARE: Extracting Experimental Findings From Clinical Literature (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing annotation schemas and datasets fail to capture real-world complexity and nuance of experimental findings. |
| Approach: | They propose a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes. |
| Outcome: | The proposed schema captures fine-grained findings as n-ary relations between entities and attributes. |
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. |
| Approach: | They propose a method that explicitly models VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve extraction accuracy. |
| Outcome: | The proposed methods show that inserting special tokens denoting layout group boundaries can lead to a 1.9% Macro F1 improvement in token classification. |
OLMES: A Standard for Language Model Evaluations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models claim to perform better on tasks measuring model capabilities, but there is no standard setup for reproducible evaluations. |
| Approach: | They propose a document that is documented and practical for reproducible LLM evaluations and includes recommendations from existing literature and new experiments. |
| Outcome: | The proposed standard identifies and reviews the varying factors in evaluation practices adopted by the community, such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation. |
MSˆ2: Multi-Document Summarization of Medical Studies (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for multi-document summarization (MDS) are either in the general domain, such as WikiSum, or very small such as DUC 1 or TAC 2011 . Existing systems for summarizing biomedical literature take 1-2 years to complete . |
| Approach: | They propose to use a multi-document summarization system based on BART to assess the quality of the summarized biomedical literature. |
| Outcome: | The proposed system has high summarization quality, but significant work remains to achieve it. |
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts (2022.emnlp-demos)
Copied to clipboard
Sonia Murthy, Kyle Lo, Daniel King, Chandra Bhagavatula, Bailey Kuehl, Sophie Johnson, Jonathan Borchardt, Daniel Weld, Tom Hope, Doug Downey
| Challenge: | Current systems that automatically define unfamiliar terms only surface a single "best" description for all users, which may not be accessible for all readers, given varying background knowledge. |
| Approach: | They propose an end-to-end system that generates sets of descriptions of scientific concepts . ACCoRD corpus includes 1,275 labeled contexts and 1,787 expert-authored concept descriptions . |
| Outcome: | The proposed system produces diverse descriptions of concepts in terms of reference concepts. |
S2abEL: A Dataset for Entity Linking from Scientific Tables (2023.emnlp-main)
Copied to clipboard
| Challenge: | Entity linking (EL) is a longstanding problem in natural language processing and information extraction. |
| Approach: | They propose a neural baseline method for EL on scientific tables containing many out-of-knowledge-base mentions and a method that significantly outperforms a generic table EL method. |
| Outcome: | The proposed method significantly outperforms state-of-the-art generic table EL method on scientific tables with many out-of knowledge-base mentions. |
SciFact-Open: Towards open-domain scientific claim verification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Current scientific claim verification systems can achieve very strong performance on limited contexts, in some cases approaching human agreement. |
| Approach: | They propose to pool and annotate top predictions from four state-of-the-art scientific claim verification models to evaluate their performance against large corpora. |
| Outcome: | The proposed system performs well on a corpus of 500K scientific abstracts. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |
KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as conversational agents. |
| Approach: | They construct a dataset of knowledge-intensive writing instructions to evaluate LLMs' ability to follow user instructions. |
| Outcome: | The proposed model fails to integrate new information into an existing answer and perform precise and unambiguous edits. |