Papers by Chantal Shaib
Detection and Measurement of Syntactic Templates in Generated Text (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing diversity evaluation focuses primarily on word-level features. |
| Approach: | They propose a method for evaluating diversity over syntactic features to characterize general repetition in large language models. |
| Outcome: | The proposed method shows that models produce templated text in downstream tasks at a higher rate than what is found in human-reference texts. |
Who Taught You That? Tracing Teachers in Model Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2006, show that model distillation can imbue efficient small language models with task-specific capabilities competitive with expensive teacher LLMs. |
| Approach: | They propose to distill outputs from a large teacher model to a small student model . they propose to use part-of-speech templates as higher-order linguistic features capable of capturing distinctive signals from teacher models that persist in distilled student outputs. |
| Outcome: | The proposed model distillation technique can imbue efficient small language models with task-specific capabilities competitive with (expensive) teacher LLMs. |
Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success) (2023.acl-short)
Copied to clipboard
| Challenge: | Large language models are capable of producing high quality summaries of general domain news articles in few- and zero-shot settings, but it is unclear whether they are similarly capable in more specialized domains such as biomedicine. |
| Approach: | They use GPT-3 to generate single- and multi-document summaries of biomedical articles, given no supervision, using a set of annotations. |
| Outcome: | The proposed model outperforms fully supervised models in generic news summarization, but struggles to synthesize evidence across multiple documents. |
Explainable Clinical Decision Support from Text (2020.emnlp-main)
Copied to clipboard
| Challenge: | Clinical prediction models often use structured variables and provide outcomes that are not readily interpretable by clinicians. |
| Approach: | They propose a hierarchical CNN-transformer model with explicit attention as an interpretable, multi-task clinical language model. |
| Outcome: | The proposed model achieves AUROCs of 0.75 and 0.78 on sepsis and mortality prediction. |
Measuring Lexical Diversity of Synthetic Data Generated through Fine-Grained Persona Prompting (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-grained personas have been used for generating ‘diverse’ synthetic data for pre-training and supervised fine-tuning of Large Language Models (LLMs). |
| Approach: | They measure the diversity of persona-driven synthetically generated prompts and responses with a suite of lexical diversity and redundancy metrics. |
| Outcome: | The proposed model is based on human-written prompts and responses, but human-generated prompts are significantly less diverse than human-created ones. |
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence (2026.findings-acl)
Copied to clipboard
Kaijie Mo, Siddhartha Venkatayogi, Chantal Shaib, Ramez Kouzy, Wei Xu, Byron C Wallace, Junyi Jessy Li
| Challenge: | Existing models are overwhelmingly accurate when presented with counterfactual medical evidence . prior work explored conflicts between context and LLM parametric knowledge in the general domain . |
| Approach: | They construct a counterfactual medical QA dataset that requires models to answer clinical comparison questions with evidence from randomized controlled trials. |
| Outcome: | The proposed model overemphasizes the latter, and the model overestimates the latter. |