| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
Similar Papers
Distributional Properties of Subword Regularization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training. |
| Approach: | They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word. |
| Outcome: | The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus. |
Investigating the Effectiveness of BPE: The Power of Shorter Sequences (D19-1)
Copied to clipboard
| Challenge: | Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, but its reasons for its effectiveness are not well understood. |
| Approach: | They link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family. |
| Outcome: | The proposed method is compared with dictionary-based compression algorithms and improves on a fixed vocabulary size budget. |
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Tokenization is a relatively understudied area, but it can greatly impact model performance and efficiency. |
| Approach: | They propose a modified BPE tokenizer that removes merges that leave intermediate "junk" tokens from the vocabulary. |
| Outcome: | The proposed method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Multilingual modelling can improve machine translation for low-resource languages, partly through shared subword representations. |
| Approach: | They propose to use subword regularisation to promote synergy and BPE to facilitate cross-lingual transfer. |
| Outcome: | The proposed methods promote synergy and prevent interference across different linguistic typologies. |
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision (2024.naacl-long)
Copied to clipboard
| Challenge: | Subword tokenisation is a mainstay in natural language processing (NLP) it allows the representation of an infinite space of text with a finite set of units. |
| Approach: | They propose to use byte-pair encoding to represent an infinite space of text with a finite set of units by removing subwords from the BPE vocabulary without impeding further use of merges that relied on them. |
| Outcome: | The proposed method improves token-based tasks in English, Dutch and German while saving training costs. |
Multi-view Subword Regularization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing subword regularization methods for multilingual pretrained representations are suboptimal for multi-lingual transfer. |
| Approach: | They propose a method that enforces consistency between standard and probabilistic segmentations. |
| Outcome: | The proposed method improves the effectiveness of cross-lingual transfer by 2.5 points over standard methods. |
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (P18-1)
Copied to clipboard
| Challenge: | Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation. |
| Approach: | They propose a method to regularize subword segmentations probabilistically by sampling subwords . they also propose 'unigram' language model to be used for better subword sampling . |
| Outcome: | The proposed method improves on low resource and out-of-domain settings with multiple corpora. |
AdaptBPE: From General Purpose to Specialized Tokenizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Subword tokenization methods impact performance and efficiency of large language models . generic tokens can incur inefficiencies when applying the model to specific domains or languages . |
| Approach: | They propose a subword tokenization technique that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. |
| Outcome: | The proposed method compresses test corpora more effectively than baselines using the same vocabulary size. |
Tokenization with Factorized Subword Encoding (2023.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization methods are often used to project subwords onto triplets . a typical tokenizer consists of 10 000s of subword mapped onto a single index . |
| Approach: | They propose a subword tokenization method that factorizes subwords onto triplets using a VQ-VAE model. |
| Outcome: | The proposed tokenization method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm. |