Papers by Byron Wallace

Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges (2023.eacl-demo)

Challenge:	Influence functions provide machinery for identifying training instances that may have led to a specific prediction, but are computationally expensive and prohibitive in many cases.
Approach:	They evaluate the degree to which different potential instance attribution agrees with respect to the importance of training samples.
Outcome:	The proposed methods exhibit desirable characteristics similar to more complex methods, but are computationally expensive.

Copied to clipboard

Sanjana Ramprasad, Jered Mcinerney, Iain Marshall, Byron Wallace

NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarization (2023.findings-eacl)

Challenge:	Existing systems that retrieve trial publications matching a query are inefficient and introduce unsupported statements.
Approach:	They propose a system that aims to automatically summarize evidence presented in the set of randomized controlled trials most relevant to a given query.
Outcome:	The proposed system retrieves trial publications matching a query specifying a combination of condition, intervention(s), and outcome(s) and ranks them according to sample size and estimated study quality.

Copied to clipboard

Junru Lu, Jiazheng Li, Byron Wallace, Yulan He, Gabriele Pergola

Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021.naacl-main)

Challenge:	a recent study shows that accessing medical literature is difficult for laypeople because it is written for specialists and contains medical jargon.
Approach:	They propose a two-stage strategy to identify relevant content to be simplified . they first generate reference summaries via sentence matching between the original and simplified abstracts .
Outcome:	The proposed approach improves on a seq2seq-based test set on an English medical corpus . it also improves the SARI score by 1.1% .

Copied to clipboard

Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, Byron Wallace

InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification (2024.acl-long)

Challenge:	Pretraining large (masked) language models over EHR data has yielded consistent performance gains across tasks.
Approach:	They propose to use large Transformers to release pretraining models over EHRs . they propose to recover patient names and conditions associated with them .
Outcome:	The proposed models recover patient names and conditions associated with patients . the proposed models share the model parameters for use by other researchers .

Copied to clipboard

Jan Trienes, Sebastian Joseph, Jörg Schlötterer, Christin Seifert, Kyle Lo, Wei Xu, Byron Wallace, Junyi Jessy Li

Paraphrasing

Challenge:	Text simplification aims to make technical texts more accessible to laypeople but often results in deletion of information and vagueness.
Approach:	They propose a framework to characterize and recover simplification-induced information loss in form of question-and-answer (QA) pairs.
Outcome:	The proposed framework characterizes and recovers simplification-induced information loss in form of question-and-answer (QA) pairs.

Disentangling Representations of Text by Masking Transformers (2021.emnlp-main)

Copied to clipboard

Xiongyi Zhang, Jan-Willem van de Meent, Byron Wallace

Semantic Text Processing

Challenge:	Large pretrained models such as BERT encode a range of features into monolithic vectors, providing strong predictive accuracy across downstream tasks.
Approach:	They explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects.
Outcome:	The proposed method disentangles sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics.

Multilingual Simplification of Medical Texts (2023.emnlp-main)

Copied to clipboard

Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vishnesh Ramanathan, Wei Xu, Byron Wallace, Junyi Jessy Li

Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success) (2023.acl-short)

Challenge:	Existing work on medical text simplification has focused on monolingual settings . important findings in medicine are typically presented in technical, jargon-laden language . text simulating models can generate viable simplified texts, but there are outstanding challenges .
Approach:	They propose a dataset for medical text simplification in four languages . they evaluate fine-tuned and zero-shot models across these languages based on human assessments and analyses .
Outcome:	The proposed dataset evaluates models in English, Spanish, French, and Farsi . it shows that the models can generate viable simplified texts, but there are challenges .

Copied to clipboard

Chantal Shaib, Millicent Li, Sebastian Joseph, Iain Marshall, Junyi Jessy Li, Byron Wallace

Revisiting Relation Extraction in the era of Large Language Models (2023.acl-long)

Challenge:	Large language models are capable of producing high quality summaries of general domain news articles in few- and zero-shot settings, but it is unclear whether they are similarly capable in more specialized domains such as biomedicine.
Approach:	They use GPT-3 to generate single- and multi-document summaries of biomedical articles, given no supervision, using a set of annotations.
Outcome:	The proposed model outperforms fully supervised models in generic news summarization, but struggles to synthesize evidence across multiple documents.

Copied to clipboard

Somin Wadhwa, Silvio Amir, Byron Wallace

Influence Functions for Sequence Tagging Models (2022.findings-emnlp)

Challenge:	Standard supervised approaches to RE learn to tag tokens comprising entity spans and then predict the relationship between them.
Approach:	They propose to use large language models for RE to evaluate their performance . they use GPT-3 and Flan-T5 large to train RE .
Outcome:	The proposed model outperforms existing models on a sequence-to-sequence task under varying levels of supervision.

Copied to clipboard

Sarthak Jain, Varun Manjunatha, Byron Wallace, Ani Nenkova

Syntactic Text Processing

Challenge:	Named Entity Recognition, Part-of-Speech tagging, and Semantic Role Labeling are standard tasks in NLP, but there has been little work on interpretability methods for sequence taging.
Approach:	They propose to extend influence functions to sequence tagging tasks by identifying noisy annotations in NER corpora.
Outcome:	The proposed methods are able to identify noisy annotations in NER corpora and are scalable.

Paragraph-level Simplification of Medical Texts (2021.naacl-main)

Copied to clipboard

Ashwin Devaraj, Iain Marshall, Byron Wallace, Junyi Jessy Li

Learning from Natural Language Explanations for Generalizable Entity Matching (2024.emnlp-main)

Challenge:	Existing methods for simplification of medical texts are limited due to jargon and technical content.
Approach:	They propose to automate the simplification of medical texts by penalizing decoders for producing "jargon" terms.
Outcome:	The proposed method improves on existing heuristics by penalizing the decoder for producing "jargon" terms.

Copied to clipboard

Somin Wadhwa, Adit Krishnan, Runhui Wang, Byron Wallace, Luyang Kong

Self-Repetition in Abstractive Neural Summarizers (2022.aacl-short)

Challenge:	Entity matching is the task of linking records from different sources that refer to the same real-world entity.
Approach:	They propose to "distill" LLM reasoning into smaller entity matching models via natural language explanations.
Outcome:	The proposed model distillation approach achieves strong performance on out-of-domain generalization tests (10.85% F-1).

Copied to clipboard

Nikita Salkar, Thomas Trikalinos, Byron Wallace, Ani Nenkova

Combining Feature and Instance Attribution to Detect Artifacts (2022.findings-acl)

Challenge:	a quantitative and qualitative analysis of self-repetition in neural summarization systems is needed to improve training.
Approach:	They propose a quantitative and qualitative analysis of self-repetition in neural summarization systems . they analyze the behavior of three popular architectures fine-tuned on five datasets .
Outcome:	The proposed model produces formulaic repetition in summaries, but not in summarization tasks.

Copied to clipboard

Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, Byron Wallace

Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study (2024.eacl-short)

Challenge:	In this paper, we evaluate use of different attribution methods for aiding identification of training data artifacts.
Approach:	They propose hybrid methods that combine saliency maps and instance attribution methods to aid in identifying training data artifacts.
Outcome:	The proposed methods can be used to efficiently uncover artifacts in training data when a challenging validation set is available.

Copied to clipboard

Zhaoyue Sun, Gabriele Pergola, Byron Wallace, Yulan He

Information Extraction & Text Mining

Challenge:	pharmacovigilance event extraction is a key field of healthcare that involves identifying, evaluating, understanding, and preventing adverse effects.
Approach:	They investigate the ability of large language models (LLMs) to extract adverse events from medical text.
Outcome:	The proposed model performs reasonably well with demonstration selection strategies, but falls short compared to fully fine-tuned small models.

RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (2023.findings-eacl)

Copied to clipboard

Somin Wadhwa, Vivek Khetan, Silvio Amir, Byron Wallace

Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews (2023.emnlp-main)

Challenge:	Social media platforms such as Reddit are vulnerable to misinformation and disinformation.
Approach:	They propose a method to automatically derive (noisy) supervision for retrieval of trustworthy evidence relevant to a given claim made on social media.
Outcome:	The proposed method outperforms baseline models in the retrieval task performed by medical doctors.

Copied to clipboard

Hye Yun, Iain Marshall, Thomas Trikalinos, Byron Wallace

Challenge:	Medical systematic reviews are time-consuming and often generate inaccurate outputs . authors: a model that generates scientific-sounding outputs can be unusable at best .
Approach:	They conduct interviews with systematic review experts to characterize perceived utility and risks of LLMs in medical evidence reviews.
Outcome:	a new study characterizes perceived utility and risks of medical evidence reviews . experts say they can assist in the writing process by drafting summaries, distilling information . authors say they expect the model to be more accurate and more reliable .

On-the-fly Definition Augmentation of LLMs for Biomedical NER (2024.naacl-long)

Copied to clipboard

Monica Munnangi, Sergey Feldman, Byron Wallace, Silvio Amir, Tom Hope, Aakanksha Naik

Low-Resource NLP

Challenge:	Despite their general capabilities, LLMs struggle on biomedicalNER tasks due to specialized terminology and lack of training data.
Approach:	They propose a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly.
Outcome:	The proposed approach improves performance on biomedicalNER tasks by 15% (on average) The proposed method outperforms fine-tuned language models in few-shot settings.

Towards Reducing Diagnostic Errors with Interpretable Risk Prediction (2024.naacl-long)

Copied to clipboard

Denis McInerney, William Dickinson, Lucy Flynn, Andrea Young, Geoffrey Young, Jan-Willem van de Meent, Byron Wallace

Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (2024.emnlp-main)

Challenge:	Diagnostic errors occur because clinicians cannot easily access relevant information in EHRs.
Approach:	They propose a method to use LLMs to identify pieces of evidence that indicate increased or decreased risk of specific diagnoses in patient EHRs.
Outcome:	The proposed method reduces diagnostic errors by identifying evidence in patient EHRs . it uses a Neural Additive Model to make predictions backed by evidence at time-points where clinicians are uncertain .

Copied to clipboard

Sheridan Feucht, David Atkinson, Byron Wallace, David Bau

Semantic Text Processing

Challenge:	Current language models process text as sequences of tokens that roughly correspond to words . individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise .
Approach:	They propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers.
Outcome:	The proposed method "reads out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers.

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature (P18-1)

Copied to clipboard

Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, Byron Wallace

Reasoning

Challenge:	In 2015 alone, about 100 manuscripts describing randomized controlled trials for medical interventions were published every day.
Approach:	They propose a corpus of 5,000 medical articles annotated with demarcations of text spans that describe the Patient population enrolled, the Interventions studied and to what they were Compared, and the Outcomes measured.
Outcome:	The proposed corpus includes 5,000 medical articles describing clinical randomized controlled trials.

On the Impact of Random Seeds on the Fairness of Clinical Classifiers (2021.naacl-main)

Copied to clipboard

Silvio Amir, Jan-Willem van de Meent, Byron Wallace

Investigating Mysteries of CoT-Augmented Distillation (2024.emnlp-main)

Challenge:	Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s).
Approach:	They explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) they find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements.
Outcome:	The proposed model fairness is based on the MIMIC-III dataset, the standard dataset in clinical NLP research.

Copied to clipboard

Somin Wadhwa, Silvio Amir, Byron Wallace

Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data (2021.emnlp-main)

Challenge:	Recent studies show that eliciting chain of thought rationales from a large "teacher" model in addition to target labels yields (often substantial) improvements in model distillation.
Approach:	They ask: Why and how does this additional training signal help in model distillation?
Outcome:	The proposed method improves model performance on question answering tasks by eliciting CoT rationales from a student model in addition to target labels.

Copied to clipboard

David Lowell, Brian Howard, Zachary C. Lipton, Byron Wallace

Low-Resource NLP

Challenge:	Unsupervised Data Augmentation (UDA) is a semisupervised learning method that penalizes differences between a model's predictions on unlabeled examples and corresponding 'noised' examples produced via data augmentation.
Approach:	They propose to use a consistency loss to penalize differences between models' predictions on unlabeled and unlabed examples to enforce consistency between models and their perturbed counterparts.
Outcome:	The proposed method is able to penalize differences between models' outputs on unlabeled and unlabed examples without complex data augmentation.

CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models (2023.findings-emnlp)

Copied to clipboard

Denis McInerney, Geoffrey Young, Jan-Willem van de Meent, Byron Wallace

PHEE: A Dataset for Pharmacovigilance Event Extraction from Text (2022.emnlp-main)

Challenge:	a recent study suggests that linear models with interpretable features are more reliable than opaque models.
Approach:	They propose an approach for natural-language specification of features for linear models . they prompt LLMs with expert-crafted queries to generate interpretable features from health records .
Outcome:	The proposed approach can be used to craft features clinically meaningful for downstream tasks . it is based on a risk prediction task and standard predictive tasks based upon this data .

Copied to clipboard

Zhaoyue Sun, Jiazheng Li, Gabriele Pergola, Byron Wallace, Bino John, Nigel Greene, Joseph Kim, Yulan He

Information Extraction & Text Mining

Challenge:	Using NLP methods to discover and extract adverse drug events from unstructured textual data is difficult because it requires time-consuming manual curation.
Approach:	They propose to use a hierarchical event schema to extract annotated events from medical case reports and biomedical literature to analyze patient data.
Outcome:	The proposed dataset is the largest public dataset to date and contains over 5000 events from medical case reports and biomedical literature.

That’s the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data (2022.emnlp-main)

Copied to clipboard

Jered McInerney, Geoffrey Young, Jan-Willem van de Meent, Byron Wallace

FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence (2024.acl-long)

Challenge:	Recent multimodal models induce soft local alignments between image regions and sentences.
Approach:	They compare alignments from a state-of-the-art multimodal model for EHR with human annotations that link image regions to sentences.
Outcome:	The proposed models induce soft local alignments between image regions and sentences . the text has an often weak or unintuitive influence on attention, the authors found .

Copied to clipboard

Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, Junyi Jessy Li

Summarization

Challenge:	FactPICO is a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials . existing metrics for factual summarizing medical evidence are poorly correlated with expert judgments on the instance level.
Approach:	They propose a factuality benchmark for plain language summarization of medical texts . they assess factuality of critical elements of RCTs in those summaries .
Outcome:	The proposed benchmark assesses the factuality of medical summaries using LLMs . the summary summators are based on 345 plain language summaires with fine-grained evaluation .

Evaluating Factuality in Text Simplification (2022.acl-long)

Copied to clipboard

Ashwin Devaraj, William Sheffield, Byron Wallace, Junyi Jessy Li

Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains (2024.eacl-short)

Challenge:	Automated simplification models aim to make input texts more readable without altering their meaning.
Approach:	They propose a taxonomy of errors that are used to analyze simplification models . they propose to use simplification methods to make input texts more readable .
Outcome:	The proposed models introduce errors that are not captured by existing evaluation metrics.

Copied to clipboard

Sanjana Ramprasad, Kundan Krishna, Zachary Lipton, Byron Wallace

Syntactic Patterns Improve Information Extraction for Medical Search (N18-2)

Challenge:	Recent work has shown that large language models can generate zero-shot summaries without explicit supervision that are often comparable or even preferred to manually composed reference summary.
Approach:	They evaluate large language models (LLMs) that generate zero-shot summaries without explicit supervision that are often comparable to manual reference summary . they acquire annotations from domain experts to identify inconsistencies in summaires and categorize errors.
Outcome:	The proposed model outperforms fine-tuned models in biomedical articles and legal bills across specialized domains.

Copied to clipboard

Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, Byron Wallace

USB: A Unified Summarization Benchmark Across Tasks and Domains (2023.findings-emnlp)

Challenge:	Medical professionals search the literature by specifying the type of patients, the medical intervention(s) and the outcome measure(s).
Approach:	They propose to exploit the availability of structured abstracts to extract medically relevant information from syntactic patterns.
Outcome:	The proposed models differ from the constituent unigrams in the extracted patterns, suggesting that they capture contextual information that is otherwise lost.

Copied to clipboard

Kundan Krishna, Prakhar Gupta, Sanjana Ramprasad, Byron Wallace, Jeffrey Bigham, Zachary Lipton