Papers by John Richardson
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (D18-2)
Copied to clipboard
| Challenge: | Existing subword segmentation tools assume input is pre-tokenized into word sequences, but SentencePiece can train subword models directly from raw sentences. |
| Approach: | They propose a language-independent subword tokenizer and detokenizer for Neural-based text processing. |
| Outcome: | The proposed system achieves comparable accuracy to training from raw sentences. |