One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown that contextualized word representations are a viable alternative to simple word prediction tasks. |
| Approach: | They propose to use subword units and characters to model morphology, syntax, and semantics instead of word embeddings. |
| Outcome: | The proposed representations are better for modeling syntax and more robust to noisy input. |
Similar Papers
When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Subword segmentation algorithms can produce sub-optimal segmentation when the target language is rich in morphological changes or there is not enough data for learning compact composition rules. |
| Approach: | They compare character-based and subword-based neural machine translation systems . they find character-driven models are better at handling morphological phenomena . |
| Outcome: | The character-based models are better at handling morphological phenomena, generating rare and unknown words, and more suitable for transferring to unseen domains. |
Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use subword vocabularies to process and generate text. |
| Approach: | They find that Large Language Models (LLMs) perform poorly at handling some types of affixations because subwords are marked as initial- or intra-word . |
| Outcome: | The largest models trained on enough data can mitigate this tendency because initial- and intra-word embeddings are aligned; in-context learning also helps when all examples are selected in a consistent way; but only morphological segmentation can achieve a near-perfect accuracy. |
Compositional Representation of Morphologically-Rich Input for Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural machine translation models are typically trained with fixed-size input and output vocabularies, which creates a bottleneck on their accuracy and generalization capability. |
| Approach: | They propose to replace the source-language embedding layer of NMT with a bi-directional recurrent neural network that generates compositional representations of the input at any desired level of granularity. |
| Outcome: | The proposed approach outperforms existing methods in a low-resource setting with five languages . the proposed approach consistently outperformed existing methods with a single word representation . |
On the Importance of Word Boundaries in Character-level Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural Machine Translation models typically use a fixed-size lexical vocabulary . subword segmentation methods rely on statistical heuristics that lack any linguistic notion . |
| Approach: | They propose a hierarchical decoding architecture for character-level NMT using subwords . they propose fewer parameters and a more efficient approach to perform translation at the level of words . |
| Outcome: | The proposed model can reach higher translation accuracy than the subword-level model with fewer parameters while maintaining longer-distance contextual and grammatical dependencies. |
Finding the Optimal Vocabulary Size for Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Class imbalance is said to exist when one or more classes are not of approximately equal frequency in data. |
| Approach: | They cast neural machine translation (NMT) as a classification task in an autoregressive setting and examine its limitations. |
| Outcome: | The proposed model performs better on multiple languages with large data sizes with different vocabulary sizes. |
Improving Neural Machine Translation by Incorporating Hierarchical Subword Features (C18-1)
Copied to clipboard
| Challenge: | Using subwords, we find that the appropriate subword units for the three layers differ depending on the model . incorporating hierarchical subword features improves BLEU scores on the IWSLT evaluation datasets. |
| Approach: | They propose a method that expresses a word by combining "subwords" they propose to incorporate hierarchical subword features into a single embedding layer . |
| Outcome: | The proposed method improves BLEU scores on the IWSLT evaluation datasets. |
Categorizing Semantic Representations for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Modern neural machine translation models suffer limitation in compositional generalization, resulting in weakened translation performance on unseen compounds. |
| Approach: | They propose to introduce categorization to the contextualized representations to improve generalization by reducing sparsity and overfitting. |
| Outcome: | The proposed method reduces compositional generalization error rates by 24% on a dedicated MT dataset. |
Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling Baselines (N19-1)
Copied to clipboard
| Challenge: | Experimental results show that standalone n-gram models lend themselves as natural choices for resource-lean or morphologically rich languages. |
| Approach: | They run experiments on 50 languages covering all morphological language families to compare n-gram models with lstm models. |
| Outcome: | The proposed extension outperforms an lstm language model on 42 languages while its extension which explicitly injects linguistic knowledge outperformed the character-aware neural model on 8 languages. |
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (D18-1)
Copied to clipboard
| Challenge: | Translating characters instead of words or word-fragments can simplify the processing pipeline but results in longer sequences . |
| Approach: | They propose to use sequence-to-sequence architectures of sufficient depth to solve the problem . they also evaluate the performance versus computation time tradeoffs they offer . |
| Outcome: | The proposed models outperform models operating over word fragments in character-level NMT, the authors show . they also show that the proposed models do not match the performance of their deep character baseline model . |
From Bytes to Subwords: Challenges of Input Representations in NLP (2026.findings-acl)
Copied to clipboard
| Challenge: | Traditionally, characters or words have been used, but recently, subwords have become the standard. |
| Approach: | They examine the current use of tokenizers and examine the weaknesses of character normalization . they propose proof of concept alternatives focused on fairness and efficiency . |
| Outcome: | The proposed model is based on a systematic review of current tokenizers and character encodings. |