Papers by Dan Pechi
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered. |
| Approach: | They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language. |
| Outcome: | The proposed model performance is correlated with pre-training perplexity and performance. |