Papers by Shuhao Gu
Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, continuous learning methods suffer from catastrophic forgetting problem, causing model to forget previous knowledge while learning new knowledge. |
| Approach: | They propose a two-stage continuous learning method based on local features of the real loss to avoid catastrophic forgetting problem. |
| Outcome: | The proposed method achieves significant improvements on domain adaptation and more challenging language adaptation tasks. |
Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mapping (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model structure and training scheme cannot ensure universal representations and cross-mappings because of lacking explicit constraints. |
| Approach: | They propose a state mover’s distance model to model the difference of the representations output by the encoder and a agreement-based training scheme to minimize the proposed distance to learn universal representations. |
| Outcome: | The proposed model can translate between languages unseen during training, i.e., zero-shot translation. |
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for domain adaptation suffer from catastrophic forgetting, large domain divergence, and model explosion. |
| Approach: | They propose a method which prunes the model and keeps the important neurons or parameters responsible for both general-domain and in-domain translation. |
| Outcome: | The proposed method improves on different language pairs and domains compared with strong baselines. |
Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models do not differentiate between semantic and linguistic features, resulting in the entanglement of knowledge and linguistics within the model. |
| Approach: | They propose to exploit both semantic and linguistic features to enhance multilingual translation by disentangling encoder representations and integrating low-level linguistic encoders. |
| Outcome: | The proposed model improves zero-shot translation while maintaining performance in supervised translation on multilingual datasets. |
Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models are usually based on attention-based encoder-decoder frameworks. |
| Approach: | They introduce a seer decoder into the encoder-decoder framework during training . they force the conventional decoded decodes to simulate the behavior of the seer . |
| Outcome: | The proposed method outperforms baselines on Chinese, English and German translation tasks. |
Improving Domain Adaptation Translation with Domain Invariant and Specific Information (N19-1)
Copied to clipboard
| Challenge: | Neural machine translation models are based on the encoder-decoder architecture, which makes them overfitting to frequent observations. |
| Approach: | They propose a method to explicitly model out-of-domain information in an encoder-decoder framework . they propose combining out- of-domain training data with out-out-of domain data . |
| Outcome: | The proposed method outperforms baselines on multiple data sets. |
Addressing the Length Bias Challenge in Document-Level Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level neural machine translation (DNMT) models incorporate context information through increased maximum lengths of source and target sentences. |
| Approach: | They propose a sliding decoding strategy that limits the length of target sentences . they propose 'length-normalized attention mechanism' to aid the model in focusing on target information . |
| Outcome: | The proposed method can achieve state-of-the-art results on open datasets. |
Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to multilingual neural machine translation tend to preserve general knowledge, but ignore language-specific knowledge. |
| Approach: | They propose to divide model neurons into general and language-specific parts based on their importance across languages. |
| Outcome: | The proposed model can preserve general knowledge but ignore language-specific knowledge on several languages, and is universal and cost-effective. |
Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation (D19-1)
Copied to clipboard
| Challenge: | Context modeling is essential to generate coherent and consistent translation for document-level Neural Machine Translations. |
| Approach: | They propose a query-guided capsule network to cluster context information into different perspectives from which the target translation may concern. |
| Outcome: | The proposed model outperforms baseline models on multiple datasets of different domains. |
Enhancing Neural Machine Translation with Semantic Units (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to model and leverage semantic units in natural language do not provide a complete understanding of the whole sentence. |
| Approach: | They propose a method which models the integral meanings of semantic units within a sentence . they propose 'word pair encoder' to help identify the boundaries of semantic unit boundaries . |
| Outcome: | The proposed method outperforms baselines and supports the semantic unit representation of subwords and tokens. |
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way (2023.emnlp-demo)
Copied to clipboard
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, Xipeng Qiu
| Challenge: | Large language models (LLMs) are increasingly pivotal in a wide range of tasks . however, the resources required for training these models necessitate efficient solutions . |
| Approach: | They propose a library that facilitates collaborative training of large language models . they use 3D parallelism, parameter-efficient fine-tuning methods and optimizers . |
| Outcome: | The proposed library has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. |
Token-level Adaptive Training for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural machine translation (NMT). |
| Approach: | They propose to assign tokens with different frequencies to target tokens during training to encourage the model to pay more attention to low-frequency tokens. |
| Outcome: | The proposed model yields consistent improvements on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens. |
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Neural machine translation models suffer from catastrophic forgetting during continual training . models tend to overfit to frequent observations in the in-domain data but forget previously learned knowledge. |
| Approach: | They investigated the causes of catastrophic forgetting in NMT models by examining their parameters and modules. |
| Outcome: | The proposed model forgets previously learned knowledge and swings to fit new data . the results show that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general- domain. |
Scaling Law for Document Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) methods fail to capture discourse phenomena such as pronominal anaphora, lexical consistency, and document coherence as the input text exceeds a single sentence. |
| Approach: | They examine the effects of model scale, data scale, and sequence length on translation quality when model size is limited. |
| Outcome: | The proposed model scales and data scales are compared with the existing models and show that increasing sequence length improves translation quality when model size is limited. |