Papers by Joel Hestness
MediSwift: Efficient Sparse Pre-trained Biomedical Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models are typically trained on general source data forvarious domains, but domain-specific pre-training is expensive and requires computational costs. |
| Approach: | They propose a suite of biomedicalLMs that leverage sparse pre-training on domain-specific biomedically text data. |
| Outcome: | The proposed model outperforms existing LLMs on biomedical tasks by 22.5x . |
Compositional Generalization for Primitive Substitutions (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to encoding compositional generalization are lacking . et al., 2017) argue that neural networks lack compositional ability . |
| Approach: | They propose a method to encode compositionality in neural networks using two representations . they reduce the entropy in each representation to improve generalization . |
| Outcome: | The proposed approach improves performance on five NLP tasks including instruction learning and machine translation. |