Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train character-level models require very deep architectures that are difficult and slow to train. |
| Approach: | They propose to fine tune a Transformer token-based model to get a model without token segmentation. |
| Outcome: | The proposed model improves translation quality and robustness to noise while requiring less token segmentation. |
Similar Papers
On the Importance of Word Boundaries in Character-level Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural Machine Translation models typically use a fixed-size lexical vocabulary . subword segmentation methods rely on statistical heuristics that lack any linguistic notion . |
| Approach: | They propose a hierarchical decoding architecture for character-level NMT using subwords . they propose fewer parameters and a more efficient approach to perform translation at the level of words . |
| Outcome: | The proposed model can reach higher translation accuracy than the subword-level model with fewer parameters while maintaining longer-distance contextual and grammatical dependencies. |
Combining Subword Representations into Word-level Representations in the Transformer Architecture (2020.acl-srw)
Copied to clipboard
| Challenge: | Currently dominant approaches use word-level tokens, but this increases the length of the sequences and makes it difficult to profit from word-based information. |
| Approach: | They propose to combine subword-level representations into word-level ones in the first layers of the encoder, reducing the effective length of the sequences in the following layers. |
| Outcome: | The proposed model maintains translation quality with no extra word-level information . it is superior to the current dominant method for incorporating word- level source language information a priori . |
Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work shows that deeper character-based neural machine translation models outperform subword-based models. |
| Approach: | They propose to investigate the ability of character-based models to learn word senses and morphological inflections and the attention mechanism in Finnish into English translation. |
| Outcome: | The character-based models outperform subword-based model in Finnish to English translation. |
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (D18-1)
Copied to clipboard
| Challenge: | Translating characters instead of words or word-fragments can simplify the processing pipeline but results in longer sequences . |
| Approach: | They propose to use sequence-to-sequence architectures of sufficient depth to solve the problem . they also evaluate the performance versus computation time tradeoffs they offer . |
| Outcome: | The proposed models outperform models operating over word fragments in character-level NMT, the authors show . they also show that the proposed models do not match the performance of their deep character baseline model . |
When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Subword segmentation algorithms can produce sub-optimal segmentation when the target language is rich in morphological changes or there is not enough data for learning compact composition rules. |
| Approach: | They compare character-based and subword-based neural machine translation systems . they find character-driven models are better at handling morphological phenomena . |
| Outcome: | The character-based models are better at handling morphological phenomena, generating rare and unknown words, and more suitable for transferring to unseen domains. |
Improving Neural Machine Translation by Incorporating Hierarchical Subword Features (C18-1)
Copied to clipboard
| Challenge: | Using subwords, we find that the appropriate subword units for the three layers differ depending on the model . incorporating hierarchical subword features improves BLEU scores on the IWSLT evaluation datasets. |
| Approach: | They propose a method that expresses a word by combining "subwords" they propose to incorporate hierarchical subword features into a single embedding layer . |
| Outcome: | The proposed method improves BLEU scores on the IWSLT evaluation datasets. |
Shallow-to-Deep Training for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that deep training is 1:4 faster than training from scratch. |
| Approach: | They propose a shallow-to-deep training method that learns deep models by stacking shallow models. |
| Outcome: | The proposed method is 1:4 faster than training from scratch and achieves BLEU scores of 30:33 and 43:29 on two translation tasks. |
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |
Low-resource neural machine translation with morphological modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for character-based and sub-word tokenization are limited to the surface forms of the words. |
| Approach: | They propose a framework-solution for modeling complex morphology in low-resource settings using a transformer architecture and beam search-based decoder. |
| Outcome: | The proposed model improves translation performance on Kinyarwanda English translation using public-domain parallel text. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |