Papers with ZWNJ
Urdu Word Segmentation using Conditional Random Fields (CRFs) (C18-1)
Copied to clipboard
| Challenge: | Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission and space insertion issues. |
| Approach: | They propose a word segmentation system for Urdu which uses a Conditional Random Field sequence modeler with orthographic, linguistic and morphological features. |
| Outcome: | The proposed system achieves an F1 score for word boundary identification and 0.85 for sub-word boundary identification tasks. |
Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT (2020.coling-main)
Copied to clipboard
| Challenge: | a set of rules are set by the regulatory body of the Persian language regarding the use of white space and ZWNJ . only a few people follow these rules in writing formal Persian, let alone the informal language. |
| Approach: | They address problems of word segmentation and zero-width non-joiner recognition in Persian . they use a macro-averaged F1 score of 92.40% on a carefully collected corpus of 500 sentences . |
| Outcome: | The proposed problem is a sequence labeling problem in Persian . it achieves a macro-averaged F1 score of 92.40% on a carefully collected corpus of 500 sentences . |