| Challenge: | Pre-trained language models such as BERT are sub-optimal at handling morphologically rich languages. |
| Approach: | They propose a two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphology in a low-resource Kinyarwanda language. |
| Outcome: | The proposed model outperforms baseline models on the low-resource morphologically rich Kinyarwanda language by 2% in F1 score and 4.3% in average score of GLUE benchmark. |
Similar Papers
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
DagoBERT: Generating Derivational Morphology with a Pretrained Language Model (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) generate derivationally complex words, but it is unclear what they learn about other aspects of language. |
| Approach: | They propose to use BERT to examine its derivational capabilities in different settings, from unmodified pretrained models to full finetuning. |
| Outcome: | The proposed model outperforms the state-of-the-art in derivation generation. |
MorphBPE: Morphology-Aware Tokenization for Efficient LLM Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Tokenization is a key design choice in modern NLP systems and a critical bottleneck for multilingual Large Language Models. |
| Approach: | They propose a tokenization extension that constrains merge operations to respect morpheme boundaries while preserving inference. |
| Outcome: | The proposed tokenization improves morphological coherence and language model cross-entropy in four languages. |
A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages (2021.eacl-srw)
Copied to clipboard
| Challenge: | Neural dependency parsing has been a success for many domains and languages, but the bottleneck of massive labelled data limits its effectiveness for low resource languages. |
| Approach: | They propose to use morphological knowledge to improve dependency parsing for morphology rich languages in a low-resource setting to perform experiments. |
| Outcome: | The proposed method achieves an average gain of 2 points (UAS) and 3.6 points (LAS) on 10 MRLs in low-resource settings. |
BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languages (2022.findings-acl)
Copied to clipboard
| Challenge: | Morphologically rich polysynthetic languages present a challenge for NLP systems due to data sparsity. |
| Approach: | They propose to use subword segmentation to reduce data sparsity in polysynthetic languages . they compare supervised and unsupervised morphological segmentation methods to Byte-Pair Encodings . |
| Outcome: | The proposed methods outperform BPEs in MT tasks for all language pairs except for Nahuatl . the proposed methods are more efficient than supervised methods, but less sparse in fusional languages. |
Morphology-Aware Multi-Granularity Representation Learning for Agglutinative Languages (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for learning low-resource agglutinative languages are limited to word and phrase levels. |
| Approach: | They propose a morphology-aware gated multi-granularity pre-training framework for agglutinative languages . framework leverages morphological knowledge and integrates a word-level encoder to capture contextual semantics . |
| Outcome: | The proposed framework improves on Mongolian and Turkish agglutinative languages . it leverages morphological knowledge and integrates tagging and segmentation to build fine-grained representations . |
AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that large pre-trained language models are not sufficient for Hebrew. |
| Approach: | They propose a large pre-trained language model for Hebrew that recovers morphological segments encoded in contextualized embedding vectors. |
| Outcome: | The proposed model obtains state-of-the-art on all tasks beyond contemporary Hebrew baselines. |
Low-resource neural machine translation with morphological modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for character-based and sub-word tokenization are limited to the surface forms of the words. |
| Approach: | They propose a framework-solution for modeling complex morphology in low-resource settings using a transformer architecture and beam search-based decoder. |
| Outcome: | The proposed model improves translation performance on Kinyarwanda English translation using public-domain parallel text. |
Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as bytepair encoding. |
| Approach: | They propose to use BERT as an example PLM to study its semantic representations of English derivatives to test their hypothesis. |
| Outcome: | The proposed model outperforms BERT on a series of semantic probing tasks. |
HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024.naacl-srw)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. |
| Approach: | They propose two hybrid architectures that combine self-attention and additive attention mechanisms with sub-layer normalization to achieve double the pretraining accuracy of a vanilla-BERT baseline. |
| Outcome: | The proposed architectures outperform BERT-base on two downstream tasks while accelerating inference. |