Papers by Rathin Singha
TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to SWS fail when encountering out-of-vocabulary tokens . lexicon driven approaches fail when dealing with out- of-vocal tokens, authors say . |
| Approach: | They propose a novel lexicon-driven shallow parser that takes into account sandhi phenomenon . they propose linguistically informed Sanskrit tokenizer module that encodes character input . |
| Outcome: | The proposed system outperforms the current state-of-the-art system by 7.2 points absolute gain on the benchmark datasets. |