Papers by Silvio Amir
Who Taught You That? Tracing Teachers in Model Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2006, show that model distillation can imbue efficient small language models with task-specific capabilities competitive with expensive teacher LLMs. |
| Approach: | They propose to distill outputs from a large teacher model to a small student model . they propose to use part-of-speech templates as higher-order linguistic features capable of capturing distinctive signals from teacher models that persist in distilled student outputs. |
| Outcome: | The proposed model distillation technique can imbue efficient small language models with task-specific capabilities competitive with (expensive) teacher LLMs. |
Revisiting Relation Extraction in the era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Standard supervised approaches to RE learn to tag tokens comprising entity spans and then predict the relationship between them. |
| Approach: | They propose to use large language models for RE to evaluate their performance . they use GPT-3 and Flan-T5 large to train RE . |
| Outcome: | The proposed model outperforms existing models on a sequence-to-sequence task under varying levels of supervision. |
RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (2023.findings-eacl)
Copied to clipboard
| Challenge: | Social media platforms such as Reddit are vulnerable to misinformation and disinformation. |
| Approach: | They propose a method to automatically derive (noisy) supervision for retrieval of trustworthy evidence relevant to a given claim made on social media. |
| Outcome: | The proposed method outperforms baseline models in the retrieval task performed by medical doctors. |
On-the-fly Definition Augmentation of LLMs for Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Despite their general capabilities, LLMs struggle on biomedicalNER tasks due to specialized terminology and lack of training data. |
| Approach: | They propose a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. |
| Outcome: | The proposed approach improves performance on biomedicalNER tasks by 15% (on average) The proposed method outperforms fine-tuned language models in few-shot settings. |
On the Impact of Random Seeds on the Fairness of Clinical Classifiers (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s). |
| Approach: | They explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) they find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements. |
| Outcome: | The proposed model fairness is based on the MIMIC-III dataset, the standard dataset in clinical NLP research. |
Investigating Mysteries of CoT-Augmented Distillation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that eliciting chain of thought rationales from a large "teacher" model in addition to target labels yields (often substantial) improvements in model distillation. |
| Approach: | They ask: Why and how does this additional training signal help in model distillation? |
| Outcome: | The proposed method improves model performance on question answering tasks by eliciting CoT rationales from a student model in addition to target labels. |
Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has shown that LLMs encode social biases and manifest in clinical tasks. |
| Approach: | They use mechanistic interpretability to uncover biases within LLMs . they find gender information is highly localized in MLP layers . |
| Outcome: | The proposed method can reveal biases and representations within LLMs in healthcare. |