Papers by Qiuhui Liu
Lipschitz Constrained Parameter Initialization for Deep Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies show that deep Transformers have difficulty in training even with residual connection and layer normalization. |
| Approach: | They propose a method that leverages the Lipschitz constraint on the initialization of Transformer parameters to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. |
| Outcome: | The proposed model outperforms previous RNN/CNN models but fails to converge with the original computation order. |
Learning Hard Retrieval Decoder Attention for Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | In this paper, we show that learning a hard retrieval attention that attends to a single token in a sentence is 1.43 times faster than the standard scaled dot-product attention. |
| Approach: | They propose a method to learn hard retrieval attention where an attention head attends to a single token in a sentence rather than all tokens. |
| Outcome: | The proposed method is 1.43 times faster in decoding while preserving translation quality on a wide range of MT tasks. |
Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies . |
| Approach: | They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships. |
| Outcome: | The proposed model can be computed in parallel and improves on the WMT 14 tasks. |
A Self-Distillation Recipe for Neural Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Neural Machine Translation (NMT) have been proven effective in improving the performance of computer vision tasks without pre-training a teacher. |
| Approach: | They propose a rank-order augmented Pearson correlation loss and an iterative distillation method to prevent the discrepancy of predictions between the student and a stronger teacher from disturbing the training. |
| Outcome: | The proposed method can lead to significant improvements over the strong Transformer baseline on low/middle/high-resource tasks, obtaining comparable or better performance with fewer layers. |
Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change (2020.acl-main)
Copied to clipboard
| Challenge: | Compared to previous studies, the performance of neural models is likely to be affected by the choice of hyper-parameters. |
| Approach: | They propose to automatically and dynamically determine batch sizes by accumulating gradients of mini-batches and performing an optimization step at just the time when the direction of gradients starts to fluctuate. |
| Outcome: | The proposed approach improves the Transformer model with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively. |
Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers (2021.naacl-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has attracted wide attention in recent years. |
| Approach: | They propose a probing-based approach to measure word translation accuracy using transformer layers. |
| Outcome: | The proposed model outperforms previous probing-based translation models. |
Modeling Task-Aware MIMO Cardinality for Efficient Multilingual Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing work has increased the modeling capacity of multilingual NMT by deepening or widening the Transformer. |
| Approach: | They propose to increase the model capacity by deepening the Transformer . they propose to use a multi-input-multi-output architecture to combine multiple inputs . |
| Outcome: | The proposed model surpasses previous work and is 1.31 times faster than existing models. |
Rewiring the Transformer with Depth-Wise LSTMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Stacking non-linear layers allows deep neural networks to model complicated functions . but residual connections within each layer fail to fuse information from previous layers effectively . |
| Approach: | They propose a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. |
| Outcome: | The proposed model improves in English-German / French and multilingual tasks with BLEU. |
Multi-Head Highly Parallelized LSTM Decoder for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | a self-attention network can be easily parallelized at sequence level, but LSTMs are slower to train . a recent study shows that LS models require a lot of computations to perform . |
| Approach: | They propose to compute LSTMs at sequence level to enable sequence-level parallelization . they use a bag-of-words representation of the preceding tokens context to approximate LStms . |
| Outcome: | The proposed model performs better than existing models while being faster to train . the model can be trained efficiently due to the highly parallelized self-attention network . |