Papers by Somin Wadhwa
Who Taught You That? Tracing Teachers in Model Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Xu et al., 2006, show that model distillation can imbue efficient small language models with task-specific capabilities competitive with expensive teacher LLMs. |
| Approach: | They propose to distill outputs from a large teacher model to a small student model . they propose to use part-of-speech templates as higher-order linguistic features capable of capturing distinctive signals from teacher models that persist in distilled student outputs. |
| Outcome: | The proposed model distillation technique can imbue efficient small language models with task-specific capabilities competitive with (expensive) teacher LLMs. |
Revisiting Relation Extraction in the era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Standard supervised approaches to RE learn to tag tokens comprising entity spans and then predict the relationship between them. |
| Approach: | They propose to use large language models for RE to evaluate their performance . they use GPT-3 and Flan-T5 large to train RE . |
| Outcome: | The proposed model outperforms existing models on a sequence-to-sequence task under varying levels of supervision. |
Learning from Natural Language Explanations for Generalizable Entity Matching (2024.emnlp-main)
Copied to clipboard
| Challenge: | Entity matching is the task of linking records from different sources that refer to the same real-world entity. |
| Approach: | They propose to "distill" LLM reasoning into smaller entity matching models via natural language explanations. |
| Outcome: | The proposed model distillation approach achieves strong performance on out-of-domain generalization tests (10.85% F-1). |
RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (2023.findings-eacl)
Copied to clipboard
| Challenge: | Social media platforms such as Reddit are vulnerable to misinformation and disinformation. |
| Approach: | They propose a method to automatically derive (noisy) supervision for retrieval of trustworthy evidence relevant to a given claim made on social media. |
| Outcome: | The proposed method outperforms baseline models in the retrieval task performed by medical doctors. |
Investigating Mysteries of CoT-Augmented Distillation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that eliciting chain of thought rationales from a large "teacher" model in addition to target labels yields (often substantial) improvements in model distillation. |
| Approach: | They ask: Why and how does this additional training signal help in model distillation? |
| Outcome: | The proposed method improves model performance on question answering tasks by eliciting CoT rationales from a student model in addition to target labels. |