Papers by Kosuke Tsujino
Language-Independent Tokenisation Rivals Language-Specific Tokenisation for Word Similarity Prediction (2020.lrec-1)
Copied to clipboard
| Challenge: | Language-independent tokenisation (LIT) methods that do not require labelled language resources or lexicons have gained popularity because of their compactness and ability to handle unseen or rare words. |
| Approach: | They empirically compare language-independent tokenisation methods with language-specific tokenisation (LST) methods using carefully created lexicons and training resources. |
| Outcome: | The proposed methods outperform LIT and LST on evaluation tasks across eight languages. |