Papers by Zebulon Goriely
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models rely on frequency information because they maximize the likelihood of tokens during training. |
| Approach: | They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. |
| Outcome: | The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. |