| Challenge: | Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission and space insertion issues. |
| Approach: | They propose a word segmentation system for Urdu which uses a Conditional Random Field sequence modeler with orthographic, linguistic and morphological features. |
| Outcome: | The proposed system achieves an F1 score for word boundary identification and 0.85 for sub-word boundary identification tasks. |
Similar Papers
A Neural CRF-based Hierarchical Approach for Linear Text Segmentation (2023.findings-eacl)
Copied to clipboard
Inderjeet Nair, Aparna Garimella, Balaji Vasan Srinivasan, Natwar Modani, Niyati Chhaya, Srikrishna Karanam, Sumit Shekhar
| Challenge: | Existing methods to segment unformatted text and transcripts explicitly train to predict segment boundaries, but they fail to provide a large annotated dataset. |
| Approach: | They propose a method to generate hierarchical segmentation structures based on Wikipedia annotations by using a neural conditional random field. |
| Outcome: | The proposed method outperforms or achieves competitive performance when compared to previous state-of-the-art algorithms. |
Syllable-based Neural Thai Word Segmentation (2020.coling-main)
Copied to clipboard
| Challenge: | Word segmentation is a challenging pre-processing step for Thai Natural Language Processing due to the lack of explicit word boundaries. |
| Approach: | They propose a neural Thai word segmenter that uses syllable embeddings to capture linguistic constraints and dilated CNN filters to capture the environment of each character. |
| Outcome: | The proposed system outperforms the previous state-of-the-art system on both in-domain and out-domain datasets. |
Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP focus on simple approaches to model the output label space . graphical models are often limited to (heuristic) greedy search and its variants . |
| Approach: | They propose an approach for efficiently training and decoding hybrids of graphical and graphical models based on Gibbs sampling. |
| Outcome: | The proposed approach improves on Dutch and Dutch with graphical models . the proposed model improves over a strong baseline on three languages . |
Neural Language Modeling for Named Entity Recognition (2020.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that named entity recognition systems are faster and more flexible for the size of the corpus. |
| Approach: | They propose to use a neural language model as an alternative to the conditional random field layer for named entity recognition. |
| Outcome: | The proposed system has a significant speed advantage with a marginal performance degradation. |
Hybrid semi-Markov CRF for Neural Sequence Labeling (P18-2)
Copied to clipboard
| Challenge: | Existing conditional random fields (CRFs) use hand-crafted features to perform sequence labeling tasks. |
| Approach: | They propose to use semi-Markov conditional random fields for neural sequence labeling in natural language processing to extract features from segments instead of words. |
| Outcome: | The proposed model achieves state-of-the-art when no external knowledge is used. |
A Probabilistic Toolkit for Multi-grained Word Segmentation in Chinese (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing tools for word segmentation are based on different linguistic theories or target different scenarios. |
| Approach: | They propose a probabilistic toolkit for multi-grained word segmentation in Chinese . they adopt semi-Markov CRF for single-grain word segmenting (SWS) . |
| Outcome: | The proposed approach can produce marginal probabilities of words during inference and significantly improve performance in the cross-domain scenario. |
CWSeg: An Efficient and General Approach to Chinese Word Segmentation (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for Chinese word segmentation have achieved state-of-the-art performance, but they pose challenges in the deployment. |
| Approach: | They propose to augment PLM-based Chinese word segmentation schemes by developing cohort training and versatile decoding strategies. |
| Outcome: | The proposed model can be used to augment existing PLM-based models and improve their performance on Chinese LLaMA and Alpaca datasets. |
Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging (N19-1)
Copied to clipboard
| Challenge: | Character-level models of tokens are effective at dealing with within-token noise and out-of-vocabulary words. |
| Approach: | They propose to eliminate the need for tokenizers by using a character-level semi-Markov conditional random field that uses neural networks for its character and segment representations. |
| Outcome: | The proposed model outperforms state-of-the-art part-of speech taggers on a noisy English dataset. |
Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision (2021.emnlp-main)
Copied to clipboard
Mieradilijiang Maimaiti, Yang Liu, Yuanhang Zheng, Gang Chen, Kaiyu Huang, Ji Zhang, Huanbo Luan, Maosong Sun
| Challenge: | Recent state-of-the-art (SOTA) effective neural network methods have been used in Chinese word segmentation (CWS) However, the robustness of the previous neural methods is limited by the large-scale annotated corpus. |
| Approach: | They propose a self-supervised Chinese word segmentation approach with a straightforward and effective architecture. |
| Outcome: | The proposed approach outperforms previous methods on 9 different CWS datasets with single criterion training and multiple criteria training and achieves better robustness. |
Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation (2021.findings-acl)
Copied to clipboard
Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Word segmentation is domain-dependent, which can be a challenge in low-resource languages like Thai and Urdu . a framework to handle out-of-domain inputs is proposed to improve word segmentation . |
| Approach: | They propose a domaingeneric domain adaptation framework and data augmentation technique to combat low-resource problems. |
| Outcome: | The proposed model outperforms the state-of-the-art Thai word segmentation method in out-of domain scenarios. |