Papers by Anton Schäfer
On the Effect of (Near) Duplicate Subwords in Language Modelling (2024.findings-acl)
Copied to clipboard
| Challenge: | Tokenisation is a core part of language models but can lead to less efficient training because it removes character-level information. |
| Approach: | They propose to use a tokenisation method to split a character sequence into subwords which are assigned random indices before being served to the LM. |
| Outcome: | The proposed model can generalise across duplicated subwords, but this incurs extra cost and is less data efficient. |