| Challenge: | Current word segmentation methods may produce different segmentations for the same strings . this occurs when strings appear in different sentences . |
| Approach: | They propose to use Japanese word segmentation methods that use a morpheme-based approach to produce different segmentations for the same strings. |
| Outcome: | The proposed method produces much more consistent segmentation than the current morpheme-based one. |
Similar Papers
Subword Segmentation in LLMs: Looking at Inflection and Consistency (2024.emnlp-main)
Copied to clipboard
| Challenge: | Subword segmentation is not linguistically guided and is not currently well understood in LLMs. |
| Approach: | They group words according to their segmentation properties and compare how well a model can solve a linguistic task for these groups using two criteria: adherence to morpheme boundaries and segmentation consistency of inflected forms of a lemma. |
| Outcome: | The results show that the criterion of segmentation consistency can predict the model’s ability to recognize and generate the lemma from an inflected form, providing evidence that subword segmentation is relevant. |
Shrinking Japanese Morphological Analyzers With Neural Networks and Semi-supervised Learning (N19-1)
Copied to clipboard
| Challenge: | Modern neural morphological analyzers consume gigabytes of memory. |
| Approach: | They propose a method which uses unigram character embeddings to train a model on labels produced by a state-of-the-art analyzer. |
| Outcome: | The proposed model outperforms dictionary-based methods in Japanese and Chinese . it uses less than 15 megabytes of space and is much smaller than the dictionary- based one . |
A unified approach to sentence segmentation of punctuated text in many languages (2021.acl-long)
Copied to clipboard
| Challenge: | Existing tools for segmenting punctuated text in many languages are limited in their language coverage and evaluation is ad hoc. |
| Approach: | They propose a new context-based modeling approach that can be trained on noisily-annotated data. |
| Outcome: | The proposed model exceeds baselines set by existing methods on English corpora and performs well on average on new multilingual evaluation set. |
Homonym normalisation by word sense clustering: a case in Japanese (2020.coling-main)
Copied to clipboard
| Challenge: | homonyms and homophones are a problem in language processing because of their distinct meanings. |
| Approach: | They propose a method that uses contextualised embeddings to cluster tokens into distinct sense groups and use these groups to normalise synonymous instances to a single representative form. |
| Outcome: | The proposed method is able to normalise synonymous instances to a single representative form in Japanese and improves on normalisation and transliteration. |
Morpheme Matters: Morpheme-Based Subword Tokenization for Korean Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing tokenizers rely on frequency-based segmentation to represent words . this often leads to inefficient token representations and oversegmentation . |
| Approach: | They propose a tokenization method that emphasizes the importance of Korean morphological structures in eojeol. |
| Outcome: | The proposed method outperforms existing tokenizers on Korean benchmark tasks and produces significantly fewer tokens per input sequence. |
Creating dialect sub-corpora by clustering: a case in Japanese for an adaptive method (L18-1)
Copied to clipboard
| Challenge: | a mixed corpus composed of different dialects is sufficiently resourced to cluster them into dialects. |
| Approach: | They propose a pipeline to derive clusters of dialects from a mixed corpus when their standard counterpart is sufficiently resourced. |
| Outcome: | The proposed pipeline can identify dialectal content when its standard counterpart is sufficiently resourced and can then cluster it into four dialects. |
Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use subword vocabularies to process and generate text. |
| Approach: | They find that Large Language Models (LLMs) perform poorly at handling some types of affixations because subwords are marked as initial- or intra-word . |
| Outcome: | The largest models trained on enough data can mitigate this tendency because initial- and intra-word embeddings are aligned; in-context learning also helps when all examples are selected in a consistent way; but only morphological segmentation can achieve a near-perfect accuracy. |
Back to Patterns: Efficient Japanese Morphological Analysis with Feature-Sequence Trie (2023.acl-short)
Copied to clipboard
| Challenge: | Accurate neural models are less efficient than non-neural models and are useless for processing billions of social media posts and handling user queries. |
| Approach: | They propose to make fast pattern-based NLP methods as accurate as possible . they propose a morphological analyzer for Japanese that induces reliable patterns . |
| Outcome: | The proposed method induces reliable patterns from a morphological dictionary and annotated data in Japanese. |
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |
Joint Word and Morpheme Segmentation with Bayesian Non-Parametric Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Language documentation often requires segmenting transcriptions of utterances into words and morphemes . a long tradition of nonparametric Bayesian models is used to handle these tasks . |
| Approach: | They propose a Bayesian model for simultaneously segmenting utterances at two levels . they use two under-resourced languages to better understand the value of weak supervision . |
| Outcome: | The proposed model can be used to identify language documents with weak supervision. |