Papers by Frank Rudzicz
Extracting relevant information from physician-patient dialogues for automated clinical note taking (D19-62)
Copied to clipboard
| Challenge: | a system that extracts pertinent medical information from dialogues between clinicians and patients is proposed . entering data into EMRs is currently slow and error-prone, and clinicians spend up to 50% of their time on data entry. |
| Approach: | They propose a system that automatically extracts medical information from dialogues between clinicians and patients using context and time information. |
| Outcome: | The proposed system extracts medical information from dialogues and automatically generates a patient note. |
Improving Automatic Quotation Attribution in Literary Novels (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quotation attribution in literary novels require varying levels of available information. |
| Approach: | They propose to train and evaluate models for character identification, coreference resolution, quotation identification and speaker attribution tasks using an annotated dataset. |
| Outcome: | The proposed model scores on speaker attribution task on the same scale as state-of-the-art models. |
Detecting dementia in Mandarin Chinese using transfer learning from a parallel corpus (N19-1)
Copied to clipboard
| Challenge: | Existing methods for automatic detection of Alzheimer's disease (AD) are limited by a lack of data. |
| Approach: | They propose a method to learn a correspondence between independently engineered lexicosyntactic features in two languages, using a large parallel corpus of out-of-domain movie dialogue data. |
| Outcome: | The proposed method outperforms both unilingual and machine translation-based baselines in Mandarin Chinese and is the first to transfer feature domains in detecting cognitive decline. |
CausalLink: An Interactive Evaluation Framework for Causal Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for causal reasoning are unclear . we propose a framework that disentangles reasoning processes from confounding factors . |
| Approach: | They propose a framework that assesses the causal reasoning skill to identify correct interventions in conversational language models. |
| Outcome: | The proposed evaluation framework isolates causal capabilities from confounding effects of world knowledge and semantic cues. |
Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document classification struggle with token limits and fail to adequately model hierarchical relationships within documents. |
| Approach: | They propose a novel model leveraging a graph-tree structure to capture local and global dependencies. |
| Outcome: | The proposed model captures syntactic relationships and broader document contexts without token limits and can handle arbitrarily long contexts. |
An Evaluation of Disentangled Representation Learning for Texts (2021.findings-acl)
Copied to clipboard
| Challenge: | Disentangled representations of texts encode information pertaining to different aspects of the text in separate vector embeddings. |
| Approach: | They propose to use a highly-structured natural language dataset to evaluate disentangled representations for texts. |
| Outcome: | The proposed models are well-suited for learning disentangled representations of texts on a synthetic natural language dataset. |
ACCORD: Closing the Commonsense Measurability Gap (2025.naacl-long)
Copied to clipboard
| Challenge: | ACCORD is a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs). |
| Approach: | They propose a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) they introduce formal elements to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. |
| Outcome: | The proposed framework can scale with future LLM improvements. |
How is BERT surprised? Layerwise detection of linguistic anomalies (2021.acl-long)
Copied to clipboard
| Challenge: | a number of studies have shown that transformer-based language models detect when a word is anomalous in context, but likelihood scores do not tell the cause of the anomaly. |
| Approach: | They propose to use Gaussian models for density estimation at intermediate layers of three language models to evaluate grammaticality. |
| Outcome: | The proposed method on BLiMP shows that language models employ different mechanisms to detect different types of linguistic anomalies. |
Predicting Fine-Tuning Performance with Probing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale neural models have recently demonstrated impressive performance in language understanding tasks, typically evaluated by their fine-tuned performance. |
| Approach: | They propose to use probing to extract a proxy signal widely used in model development to predict fine-tuning performance. |
| Outcome: | The proposed method predicts fine-tuning performance with errors 40% - 80% smaller than baselines. |
Learning multiview embeddings for assessing dementia (D18-1)
Copied to clipboard
| Challenge: | In 2017, 5.7 million Americans were living with Alzheimer's disease (AD), and the disease accounted for $11.4 billion in healthcare costs in the United States. |
| Approach: | They leverage the multiview nature of a small AD dataset to learn an embedding that captures different modes of cognitive impairment. |
| Outcome: | The proposed embeddings achieve an F1 score of 0.82 and a mean absolute error of 3.42 in the classification task and predicting clinical scores. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |
Is This LLM Library Learning? Evaluation Must Account For Compute and Behaviour (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in the coding, reasoning, and tool-use ability of LLMs have raised the possibility of library learning with LLM. |
| Approach: | They propose to use reusable and composable functions and tools to create reusable, composesable code and tools that can be reused by modifying relevant examples. |
| Outcome: | The proposed system fails to consistently outperform the baseline model and does not correct for the difference in computational cost. |
An unsupervised framework for tracing textual sources of moral change (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on moral sentiment classification and temporal inference of moral sentiment have not quantified the origins of these changes. |
| Approach: | They propose an unsupervised framework for tracing textual sources of moral change toward entities through time. |
| Outcome: | The proposed framework captures fine-grained human moral judgments and identifies coherent source topics of moral change triggered by historical events. |
Explainable Clinical Decision Support from Text (2020.emnlp-main)
Copied to clipboard
| Challenge: | Clinical prediction models often use structured variables and provide outcomes that are not readily interpretable by clinicians. |
| Approach: | They propose a hierarchical CNN-transformer model with explicit attention as an interpretable, multi-task clinical language model. |
| Outcome: | The proposed model achieves AUROCs of 0.75 and 0.78 on sepsis and mortality prediction. |
Neural reality of argument structure constructions (2022.acl-long)
Copied to clipboard
| Challenge: | lexicalist linguistic theories assume argument structure is predictable from meaning of verbs . construction grammarians propose argument structure constructions distinct from verbs. |
| Approach: | They adapt psycholinguistic studies to probe for the existence of argument structure constructions in Transformer-based language models. |
| Outcome: | The proposed method could be used to probe argument structure constructions in LMs . the study shows that LM learners prefer grouping by construction over verb grouping . |
Identification of Primary and Collateral Tracks in Stuttered Speech (2020.lrec-1)
Copied to clipboard
| Challenge: | Disfluency detection is a challenging task because of its different metrics depending on whether the input features are text or speech. |
| Approach: | They propose a framework for disfluency detection inspired by the clinical and the natural language processing perspective together with the theory of performance from (Clark, 1998) . they present a forced-aligned disfluence dataset and propose new audio features inspired by word-based span features. |
| Outcome: | The proposed framework outperforms baselines for speech-based predictions on a forced-aligned disfluency dataset from semi-directed interviews. |
Lexical Features Are More Vulnerable, Syntactic Features Have More Predictive Power (D19-55)
Copied to clipboard
| Challenge: | Existing metrics to quantify lexical diversity have been proposed. |
| Approach: | They propose to examine how generic language characteristics are impacted by text alterations. |
| Outcome: | The proposed models show that lexical features are more sensitive to text modifications than syntactic ones. |
When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation (2026.eacl-long)
Copied to clipboard
Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Prathiba Dhanesh, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
| Challenge: | Existing benchmarks for large language models are limited in scale, authenticity, and reliability due to the emotionally complex nature of therapeutic dialogue. |
| Approach: | They propose two benchmarks that provide a framework for evaluating large language models for mental health support. |
| Outcome: | The proposed framework provides a framework for generation and evaluation of large-scale authentic dialogue datasets and judge-reliability assessments. |
Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for ICD indexing have a heavy label distribution and a manual process . Xie and Xing (2017) propose a new approach to ICD re-ranking . |
| Approach: | They propose a "retrieve and re-rank" framework to allocate subsets of ICD codes to medical records . they leverage auxiliary knowledge of the electronic health records (EHR) and a discrete retrieval method . |
| Outcome: | The proposed method achieves state-of-the-art performance on the MIMIC-III benchmark. |
On the data requirements of probing (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to probe neural networks are expensive and require large datasets. |
| Approach: | They propose a method to estimate the required number of data samples in probing datasets . they use a classification task to encode a text with a deep neural network . |
| Outcome: | The proposed method estimates the required number of data samples in two probing configurations and proves it is statistically powerful. |
Auxiliary Knowledge-Induced Learning for Automatic Multi-Label Medical Document Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for ICD indexing use machine learning to assign subset of codes to medical records . experimental results show proposed method achieves state-of-the-art performance on a number of measures. |
| Approach: | They propose a method that uses a deep dilated residual convolution encoder to learn document representations across different lengths of the texts. |
| Outcome: | The proposed method achieves state-of-the-art performance on a number of measures. |
A State-Vector Framework for Dataset Effects (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent DNN-based systems gain linguistic abilities on multiple levels ranging from syntax, semantics, and even some discourse-related abilities. |
| Approach: | They propose a state-vector framework that uses idealized probing test results as the bases of a vector space to quantify the effects of both standalone and interacting datasets. |
| Outcome: | The proposed framework allows to quantify the effects of both standalone and interacting datasets. |
An information theoretic view on selecting linguistic probes (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP tasks require a question of how much linguistic knowledge is encoded in neural networks. |
| Approach: | They propose to use diagnostic classifiers to perform supervised classification from internal representations. |
| Outcome: | Empirically, the two proposed criteria lead to results that agree with each other. |
Word class flexibility: A deep contextualized approach (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on word class flexibility have been fraught with difficulties in quantifying it accurately and at scale. |
| Approach: | They propose a method to quantify word class flexibility in 37 languages using contextualized word embeddings. |
| Outcome: | The proposed method builds on recent work in contextualized word embeddings to quantify semantic shift between word classes and uncovers shared tendencies in class flexibility across languages. |
MeSHup: Corpus for Full Text Biomedical Document Indexing (2022.lrec-1)
Copied to clipboard
| Challenge: | Medical Subject Heading (MeSH) indexing is a problem of assigning a given biomedical document with the most relevant labels from an extremely large set of MeSH terms. |
| Approach: | They train an end-to-end model that combines features from documents and associated labels on MEDLINE corpus and report the new baseline. |
| Outcome: | The proposed system can be used to assign a biomedical document with the most relevant labels from an extremely large set of MeSH terms. |
Augmenting word2vec with latent Dirichlet allocation within a clinical application (N19-1)
Copied to clipboard
| Challenge: | Existing models that combine latent Dirichlet allocation and word embedding for distinguishing between speakers with and without Alzheimer’s disease from transcripts of picture descriptions are not suitable for clinical binary text classification tasks. |
| Approach: | They propose three models that combine latent Dirichlet allocation and word embedding for distinguishing between speakers with and without Alzheimer’s disease from transcripts of picture descriptions. |
| Outcome: | The proposed models outperform word2vec and LDA models on a clinical binary text classification task. |
Detecting cognitive impairments by agreeing on interpretations of linguistic features (N19-1)
Copied to clipboard
| Challenge: | Linguistic features have shown promising applications for detecting cognitive impairments. |
| Approach: | They propose a framework to classify after reaching agreements between modalities by using linguistic features to divide linguistic subsets into subset and let neural networks learn low-dimensional representations that agree with each other. |
| Outcome: | The proposed framework outperforms existing classifiers using all of the 413 linguistic features. |
Not Lost After All: How Cross-Encoder Attribution Challenges Position Bias Assumptions in LLM Summarization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Position bias is a key limitation in automatic summarization. |
| Approach: | They propose a cross-encoder-based alignment method that processes summary-source sentence pairs . |
| Outcome: | The proposed method allows better identification of semantic correspondences even when summaries substantially rewrite the source. |
KenMeSH: Knowledge-enhanced End-to-end Biomedical Text Labelling (2022.acl-long)
Copied to clipboard
| Challenge: | Medical Subject Headings (MeSH) are manually assigned to every biomedical article to facilitate retrieval of relevant information. |
| Approach: | They propose a model that combines new text features with a dynamic knowledge-enhanced mask attention that integrates document features with MeSH label hierarchy and journal correlation features to index MeSH terms. |
| Outcome: | The proposed model achieves state-of-the-art on a number of measures. |
Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025.emnlp-main)
Copied to clipboard
Yinuo Wang, Baiyang Wang, Robert Mercer, Frank Rudzicz, Sudipta Singha Roy, Pengjie Ren, Zhumin Chen, Xindi Wang
| Challenge: | achieving comprehensive trustworthiness in medical QA poses significant challenges due to complexity of healthcare data, critical nature of clinical scenarios, and multifaceted dimensions of trustworthy AI. |
| Approach: | They examine six key dimensions of trustworthiness in medical QA . they compare how each dimension is evaluated in existing LLM-based systems . |
| Outcome: | The findings show that large language models have improved patient safety and effectiveness . the models exhibit critical trust failures when deployed in clinical settings . |
Immunization against harmful fine-tuning attacks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models are often trained with safety guards to prevent harmful text generation. |
| Approach: | They propose a formal framework based on the training budget of an attacker to validate defenses against harmful fine-tuning attacks. |
| Outcome: | The proposed framework validates whether a model has been fine-tuned against harmful fine-uning attacks on harmful datasets. |
Multilingual prediction of Alzheimer’s disease through domain adaptation and concept-based language modelling (N19-1)
Copied to clipboard
Kathleen C. Fraser, Nicklas Linz, Bai Li, Kristina Lundholm Fors, Frank Rudzicz, Alexandra König, Jan Alexandersson, Philippe Robert, Dimitrios Kokkinakis
| Challenge: | Existing work on speech and language models has been limited by the size of available datasets. |
| Approach: | They propose to augment a small French dataset with a much larger English dataset to augment the language model to model the order in which information units are produced by dementia patients and controls. |
| Outcome: | The proposed model improves classification performance in English and French separately. |
Long-form evaluation of model editing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of model editing only use the ‘next few tokens’ completions after a prompt. |
| Approach: | They propose a new evaluation protocol that measures the efficacy and impact of model editing in long-form generative settings by using a machine-rated survey and a classifier which correlates well with human ratings. |
| Outcome: | The proposed evaluation protocol has little relationship with short-form metrics despite being designed to extend efficacy, generalization, locality, and portability into a long-form setting. |