Papers by Noriko Kawahara
Sudachi: a Japanese Tokenizer for Business (L18-1)
Copied to clipboard
| Challenge: | Lack of token unit compatibility is one of the critical problems of Japanese language resources. |
| Approach: | They develop a Japanese tokenizer called Sudachi and its accompanying dictionary . they use multi-granular output and normalization of notation variations to improve tokenization . |
| Outcome: | The proposed tokenizer and dictionary improve tokenization in Japanese for business use. |