| Challenge: | Existing models operate over subword tokens, but byte-based models employ a different approach . a one-hot representation of each byte does not hurt performance, but it improves BLEU scores . |
| Approach: | They propose to represent every computerized text as a sequence of bytes via UTF-8 . this eliminates the need for an embedding layer and improves performance . |
| Outcome: | The proposed model improves BLEU scores on byte-to-byte translation models compared to character-level models . the proposed model does not require an embedding layer and does not drop out of the decoder . |
Similar Papers
From Bytes to Subwords: Challenges of Input Representations in NLP (2026.findings-acl)
Copied to clipboard
| Challenge: | Traditionally, characters or words have been used, but recently, subwords have become the standard. |
| Approach: | They examine the current use of tokenizers and examine the weaknesses of character normalization . they propose proof of concept alternatives focused on fairness and efficiency . |
| Outcome: | The proposed model is based on a systematic review of current tokenizers and character encodings. |
Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent studies using pretrain-finetuning approach have achieved state-of-the-art (SOTA) performance in many natural language processing tasks. |
| Approach: | They propose a new tokenization method that combines morphology-considered subword tokenization and vocabulary methods to address this limitation. |
| Outcome: | The proposed method can be used without modifying the model structure. |
Local Byte Fusion for Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing NLP models rely on a pre-built subword tokenizer to tokenize a sentence . this can be rigid and subwords from low-resource languages are under-represented . |
| Approach: | They propose a method for byte-based machine translation that aggregates local semantic information. |
| Outcome: | The proposed method improves on multilingual translation and cross-lingual transfer . it is parameter-efficient and performs competitively to subword models, it is shown . |
Improving Neural Machine Translation by Incorporating Hierarchical Subword Features (C18-1)
Copied to clipboard
| Challenge: | Using subwords, we find that the appropriate subword units for the three layers differ depending on the model . incorporating hierarchical subword features improves BLEU scores on the IWSLT evaluation datasets. |
| Approach: | They propose a method that expresses a word by combining "subwords" they propose to incorporate hierarchical subword features into a single embedding layer . |
| Outcome: | The proposed method improves BLEU scores on the IWSLT evaluation datasets. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Where are we Still Split on Tokenization? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Identifying tokens is a crucial first step for many tasks in Natural Language Processing (NLP) gold tokenization is often assumed, but some work on token-level tasks is more challenging. |
| Approach: | They propose an efficient method for tokenization with subword-based language models and evaluate it on 122 languages in 20 scripts. |
| Outcome: | The proposed method performs on par with the state-of-the-art on 122 languages in 20 scripts. |
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (P18-1)
Copied to clipboard
| Challenge: | Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation. |
| Approach: | They propose a method to regularize subword segmentations probabilistically by sampling subwords . they also propose 'unigram' language model to be used for better subword sampling . |
| Outcome: | The proposed method improves on low resource and out-of-domain settings with multiple corpora. |
Improving Low Compute Language Modeling with In-Domain Embedding Initialisation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train language models on in-domain data are limited. |
| Approach: | They propose to initialise and freeze in-domain embeddings to provide a useful representation of rare words in English . they find that the standard configuration is not optimal when rare words are present . |
| Outcome: | The proposed approach improves language modeling by providing a useful representation of rare words in English. |
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks where large-scale parallel corpora cannot be obtained. |
| Approach: | They perform five sets of experiments to analyze when pre-trained word embeddings can be useful in NMT tasks. |
| Outcome: | The embeddings provide gains of up to 20 BLEU points in the most favorable setting. |
Reusing Weights in Subword-Aware Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | a statistical language model assigns a probability to a sequence of words . data sparsity is a major problem in building traditional n-gram language models . |
| Approach: | They propose several ways to reuse subword embeddings and other weights in subword-aware neural language models. |
| Outcome: | The proposed techniques do not benefit a competitive character-aware model . but they show significant reductions in model sizes and performance. |