Papers by Stephen Mussmann
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent datasets heuristically choose examples to ensure label balance . state-of-the-art models trained on QQP and WikiQA have only 2.4% average precision . |
| Approach: | They show that recent datasets heuristically choose examples to ensure label balance . they instead use active learning to retrieve uncertain points from a large pool of unlabeled utterance pairs . |
| Outcome: | The proposed model improves on QQP and WikiQA by using more informative negative examples. |
The price of debiasing automatic metrics in natural language evalaution (P18-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate natural language systems are expensive and expensive. |
| Approach: | They propose to combine automatic metrics with human judgment to obtain an unbiased estimator at lower cost than human evaluation alone. |
| Outcome: | The proposed estimator reduces the cost of evaluating summarization and open-response questions by 7-13%. |
An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
Gantavya Bhatt, Yifang Chen, Arnav Das, Jifan Zhang, Sang Truong, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Du, Kevin Jamieson, Jordan Ash, Robert Nowak
| Challenge: | Supervised finetuning (SFT) on instruction datasets has shown immense potential in improving the zero-shot generalization capabilities observed in large language models (LLMs). |
| Approach: | They propose to use experimental design to minimize the computational cost of active learning by identifying useful subsets of samples to annotate from an unlabeled pool. |
| Outcome: | The proposed methods save 50% of the annotation cost compared to random sampling on generative tasks. |