Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to compress Transformer are limited to sub-components, e.g., selfattention networks or embedding layer. |
| Approach: | They propose a Hybrid Tensor-Train decomposition which retains full rank and meanwhile reduces operations and parameters. |
| Outcome: | The proposed model outperforms light-weight SOTA methods on three translation tasks and achieves 7.1 points absolute improvement in BLEU and 1.27 X speedup on IWSLT’14 De-En task. |
Similar Papers
Fast and Accurate Neural Machine Translation with Translation Memory (2021.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge demonstrates the superiority of TM-based neural machine translation only on TM specialized tasks . |
| Approach: | They propose a translation memory-based approach to machine translation using a single bilingual sentence as its TM. |
| Outcome: | The proposed approach surpasses baselines on two general tasks and improves on the TM-specialized translation tasks. |
Syntactically Supervised Transformers for Faster Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Standard decoders for neural machine translation generate a single token per timestep, which slows inference . a series of controlled experiments demonstrates that SynST decodes sentences 5x faster than the baseline autoregressive Transformer. |
| Approach: | They propose a syntactically supervised Transformer that generates all target tokens in one shot . synST is a variant of the Transformer architecture that autoregressively predicts a chunked parse tree . |
| Outcome: | The proposed method decodes sentences 5x faster than the baseline method on En-De and En-Fr datasets while achieving higher BLEU scores. |
Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT). |
| Approach: | They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors. |
| Outcome: | The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets. |
The Green KNIGHT: Green Machine Translation with Knowledge-Distilled, Narrow, Inexpensive, Greedy, Hybrid Transformers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art neural machine translation models deliver high-quality translations at the expense of high inference latency and energy consumption. |
| Approach: | They propose a hardware-agnostic collection of recipes to optimize translation speed and energy consumption. |
| Outcome: | The Green KNIGHT optimizes translation speed and energy consumption with a moderate trade-off in quality. |
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |
Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive translation (NAT) is less robust in decoding batch size and hardware settings than NAT. |
| Approach: | They propose a two-stage translation prototype that prompts a small number of AT predictions and fills in previously skipped tokens at once. |
| Outcome: | The proposed translation prototype achieves comparable translation quality with AT while having 1.5x faster inference speed regardless of batch size and device. |
Multi-split Reversible Transformers Can Enhance Neural Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Large-scale transformers have been shown to improve neural machine translation performance but training these wider and deeper networks could be extremely memory intensive. |
| Approach: | They propose a multi-split based reversible transformer and a backpropagation algorithm that does not need to store activations for most layers. |
| Outcome: | The proposed model outperforms the vanilla transformer by at least 1.4 BLEU points in three datasets. |
Fully Quantized Transformer for Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art neural machine translation methods use huge amounts of parameters. |
| Approach: | They propose an all-inclusive quantization strategy for the Transformer to reduce computational costs and improve translation quality. |
| Outcome: | The proposed method achieves state-of-the-art results on most tasks compared to previous methods . |
Factorized Transformer for Multi-Domain Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-domain Neural Machine Translation (MMT) is a challenging task due to the extreme diversity of cross-domain wording and phrasing style, and the imperfections of training data distribution. |
| Approach: | They propose a factorized NMT model that divides domain-shared knowledge into domain-specific ones that are private for each constituent domain. |
| Outcome: | The proposed model achieves state-of-the-art performance and opens up new perspectives for multi-domain and open-domain applications. |
Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, Dongsoo Lee
| Challenge: | Quantization is an effective technique to address heavy computation load and memory overhead during inference. |
| Approach: | They propose a low-bit quantization strategy to represent Transformer weights by an extremely low number of bits. |
| Outcome: | The proposed model achieves 11.8 smaller model size than baseline model, with less than -0.5 BLEU. |