Papers by Anton Kolonin
Unsupervised Tokenization Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Unsupervised language learning has attracted great attention in recent years . Glushchenko et al. (2015) suggested using "deep patterns" with hierarchical "symbolic" grammatical pattern structures learned from texts as a way to model grammars and domain ontologies for natural languages. |
| Approach: | They propose to use a "transition freedom" metric to measure unsupervised tokenization . they find that different languages require different offshoots of that metric for tokenization. |
| Outcome: | The proposed method provides better tokenization quality than or comparable to lexicon-based ones, depending on the language. |