Papers by Yanjin He
Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models typically fix a vocabulary size in advance, then use Byte Pair Encoding (BPE) to construct the tokenizer. |
| Approach: | They propose a method for determining the vocabulary size by analyzing token frequency distributions through Zipf’s law and propose to use it to optimize model performance. |
| Outcome: | The proposed method improves model efficiency and effectiveness across NLP, genomics, and chemistry. |