Papers by Shuming Ma

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022.acl-long)

Challenge:	Named entity recognition (NER) suffers from the scarcity of annotated training data, especially for low-resource languages without labeled data.
Approach:	They propose a cross-lingual entity projection framework to enable zero-shot cross-linguistic NER with the help of a multilingual labeled sequence translation model.
Outcome:	The proposed method outperforms the baseline method on two benchmarks by a large margin of +3 7 F1 scores and achieves state-of-the-art performance.

Copied to clipboard

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei

Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders (2021.emnlp-main)

Challenge:	ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive .
Approach:	They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora .
Outcome:	The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost.

Copied to clipboard

Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)

Challenge:	Existing work on improving cross-lingual transferability of NMT model is under-explored.
Approach:	They propose a model that leverages a multilingual pretrained encoder to improve cross-lingual transferability.
Outcome:	The proposed model outperforms mBART and m2m-100 on a zero-shot cross-lingual transfer task.

Copied to clipboard

Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, Ryan Cotterell

Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction (D19-51)

Challenge:	Several recent papers claim to have achieved human parity at sentence-level machine translation.
Approach:	They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures.
Outcome:	The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations .

Copied to clipboard

Deli Chen, Shuming Ma, Keiko Harimoto, Ruihan Bao, Qi Su, Xu Sun

Information Extraction & Text Mining

Challenge:	Existing studies on forex prediction ignore related text completely and focus on forex trade data only, which loses important semantic information.
Approach:	They propose a BERT-based Hierarchical Aggregation Model to summarize forex news . they group news from different aspects and extract the most crucial news in each group .
Outcome:	The proposed model outperforms baseline methods and grouping methods and summarizes the influence patterns for forex trading.

PAEG: Phrase-level Adversarial Example Generation for Neural Machine Translation (2022.coling-1)

Copied to clipboard

Juncheng Wan, Jian Yang, Shuming Ma, Dongdong Zhang, Weinan Zhang, Yong Yu, Zhoujun Li

Multilingual Agreement for Multilingual Neural Machine Translation (2021.acl-short)

Challenge:	Existing methods for adversarial example generation are word-level or character-level, which ignore the ubiquitous phrase structure.
Approach:	They propose a phrase-level adversarial example generation framework to enhance the robustness of the translation model by adopting a sentence-level substitution strategy.
Outcome:	The proposed method improves translation performance and robustness to noise on three benchmarks.

Copied to clipboard

Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021.emnlp-main)

Challenge:	Existing models that only use auxiliary languages to encourage multilingual agreement ignore the relationships between different language pairs.
Approach:	They propose a multilingual agreement-based method which explicitly models the agreement between different translation directions by randomly substituting some fragments of the source language with their counterpart translations of auxiliary languages.
Outcome:	The proposed method improves on the multilingual translation task of 10 language pairs.

Copied to clipboard

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, Furu Wei

Smart-Start Decoding for Neural Machine Translation (2021.naacl-main)

Challenge:	Multilingual T5 pretrains a sequence-to-sequence model on monolingual texts, but it has shown promising results on many cross-lingual tasks.
Approach:	They propose a partially non-autoregressive objective for text-to-text pre-training and propose mT6 to improve cross-lingual transferability over multilingual T5.
Outcome:	The proposed model improves cross-lingual transferability over existing models.

Copied to clipboard

Jian Yang, Shuming Ma, Dongdong Zhang, Juncheng Wan, Zhoujun Li, Ming Zhou

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023.acl-long)

Challenge:	Existing neural machine translation models adopt a monotonic decoding order of either left-to-right or right-to left.
Approach:	They propose a method that starts decoding target words from the right side of a median word and generates words on the left.
Outcome:	The proposed method outperforms baseline models on three datasets.

Copied to clipboard

Jian Yang, Shuming Ma, Li Dong, Shaohan Huang, Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, Zhoujun Li

A Simple and Effective Unified Encoder for Document-Level Machine Translation (2020.acl-main)

Challenge:	Existing pre-training methods underutilize the benefits of language understanding for generation.
Approach:	They propose a GAN-style model for encoder-decoder pre-training with an auxiliary discriminator.
Outcome:	The proposed model outperforms existing pre-trained models and achieves state-of-the-art performance.

Copied to clipboard

Shuming Ma, Dongdong Zhang, Ming Zhou

Deconvolution-Based Global Decoding for Neural Machine Translation (C18-1)

Challenge:	Existing models for document-level machine translation use two separate encoders to model the source sentences and document- level contexts.
Approach:	They propose a unified encoder that can outperform existing models of dual-encoder models . they propose to use document-level contexts to model the interaction between the contexts and the source sentences .
Outcome:	The proposed model outperforms baseline models of dual-encoder models in terms of BLEU and METEOR scores.

Copied to clipboard

Junyang Lin, Xu Sun, Xuancheng Ren, Shuming Ma, Jinsong Su, Qi Su

Machine Translation

Challenge:	Existing models for Neural Machine Translation (NMT) use Recurrent Neural Network (RNN) to generate translation word by word following a sequential order.
Approach:	They propose a Neural Machine Translation (NMT) model that decodes the sequence with the guidance of its structural prediction of the target-side context.
Outcome:	The proposed model is more competitive compared with the state-of-the-art methods and reduces repetition with the instruction from the target-side context for decoding.

Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)

Copied to clipboard

Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi, Shiguo Lian

TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity.
Approach:	They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations.
Outcome:	The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization.

Copied to clipboard

Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, Zhaochuan Gao, Anthony Aue, Arul Menezes, Furu Wei

Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022.emnlp-main)

Challenge:	Existing approaches to multilingual sequence-to-sequence pre-training rely on monolingual corpora and sometimes synthetic document-level bilingual corporata.
Approach:	They propose to leverage document-level trilingual parallel corpora to improve sequence-to-sequence multilingual pre-training by using a novel method called Grafting.
Outcome:	The proposed method achieves strong state-of-the-art (SOTA) scores on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark.

Copied to clipboard

Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, Houfeng Wang

Improving Neural Machine Translation with Soft Template Prediction (2020.acl-main)

Challenge:	Existing work focuses on monolingual prompts, but we study multilingual prompt for multilingual models.
Approach:	They propose a model that uses a unified prompt for all languages, called UniPrompt, to alleviate the effort of designing different prompts for multiple languages.
Outcome:	The proposed model outperforms baseline models in the zero-shot cross-lingual setting.

Copied to clipboard

Jian Yang, Shuming Ma, Dongdong Zhang, Zhoujun Li, Ming Zhou

Global Encoding for Abstractive Summarization (P18-2)

Challenge:	Recent advances in neural machine translation (NMT) depend on source text to generate translation.
Approach:	They propose to use extracted templates from tree structures as soft target templates to guide the translation procedure.
Outcome:	The proposed model outperforms baseline models on four benchmarks and demonstrates the effectiveness of soft target templates.

Copied to clipboard

Junyang Lin, Xu Sun, Shuming Ma, Qi Su

SGM: Sequence Generation Model for Multi-label Classification (C18-1)

Challenge:	Existing models for abstractive summarization suffer from repetition and semantic irrelevance.
Approach:	They propose a global encoding framework which controls the information flow from the encoder to the decoder based on the global information of the source context.
Outcome:	The proposed model outperforms baseline models on the LCSTS and English Gigaword and can generate summary of higher quality and reduce repetition.

Copied to clipboard

Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, Houfeng Wang

Information Retrieval

Challenge:	Existing methods ignore the correlations between labels and different parts of the text can contribute differently for predicting different labels.
Approach:	They propose to view the multi-label classification task as a sequence generation problem and apply a decoder-based sequence generation model to solve it.
Outcome:	The proposed methods outperform previous work by a substantial margin.

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers (2023.findings-acl)

Copied to clipboard

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, Furu Wei

StableMoE: Stable Routing Strategy for Mixture of Experts (2022.acl-long)

Challenge:	Large pretrained language models have shown surprising in-context learning ability . despite the great success in performance, its working mechanism remains unclear .
Approach:	They explain language models as meta-optimizers and understand in-context learning as implicit finetuning . they find that Transformer attention has a dual form of gradient descent .
Outcome:	The proposed model can predict labels for unseen inputs without parameter updates . the proposed model outperforms smaller models with a single parameter update .

Copied to clipboard

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

Challenge:	Existing learning-to-route methods suffer from the routing fluctuation issue . with the model scale growing, training speed will go slower and memory requirements are heavy .
Approach:	They propose a Mixture-of-Experts technique that can scale up the model size of Transformers with an affordable computational overhead.
Outcome:	The proposed method outperforms existing learning-to-route methods on language modeling and multilingual machine translation.

Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation (P19-1)

Copied to clipboard

Shuming Ma, Pengcheng Yang, Tianyu Liu, Peng Li, Jie Zhou, Xu Sun

Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation (2022.acl-long)

Challenge:	Existing methods for table-to-text generation use encoder-decoder framework, but lack of large parallel data is a problem for many domains.
Approach:	They propose a model to separate table-to-text generation into two stages: key fact prediction and surface realization.
Outcome:	The proposed model achieves 27.34 BLEU score with only 1,000 parallel data, while the baseline model only achieves 9.71 BLUE score.

Copied to clipboard

Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei

On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation (2023.findings-acl)

Challenge:	Existing unsupervised neural machine translation systems can degrade when labeled data is limited.
Approach:	They propose a multilingual pretraining and multilingual fine-tuning for facilitating cross-lingual transfer in zero-shot translation using a parallel dataset.
Outcome:	The proposed model outperforms state-of-the-art models on many-to-English translation by over 7.2 and 5.0 BLEU.

Copied to clipboard

Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, Baobao Chang

Semantic-Unit-Based Dilated Convolution for Multi-Label Text Classification (D18-1)

Challenge:	Despite its success, multilingual neural machine translation suffers from the off-target issue, where the translation is in the wrong language.
Approach:	They propose a language-aware vocabulary sharing algorithm that can be used to increase the lexical distance between languages by isolating the vocab of different languages in the decoder.
Outcome:	The proposed algorithm reduces off-target rate for 90 translation tasks from 29% to 8%, while improving overall BLEU score by an average of 1.9 points without extra training cost or sacrificing the supervised directions’ performance.

Copied to clipboard

Junyang Lin, Qi Su, Pengcheng Yang, Shuming Ma, Xu Sun

Information Extraction & Text Mining

Challenge:	a novel model for multi-label text classification is proposed for the task of assigning multiple labels for a given text.
Approach:	They propose a novel model for multi-label text classification based on sequence-to-sequence learning and a hybrid attention mechanism that extracts both the word-level and the semantic unit.
Outcome:	The proposed model is competitive to the baseline models and more robust to classifying low-frequency labels.

Bag-of-Words as Target for Neural Machine Translation (P18-2)

Copied to clipboard

Shuming Ma, Xu Sun, Yizhong Wang, Junyang Lin

Machine Translation

Challenge:	Existing neural machine translation models only use one correct sentence as the target, and the other correct sentences are punished as the incorrect ones.
Approach:	They propose an approach that uses both the sentences and the bag-of-words as targets in the training stage to encourage the model to generate the potentially correct sentences that are not appeared in the train set.
Outcome:	The proposed model outperforms baseline models on a Chinese-English translation dataset by the BLEU score of 4.55.

Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation (N18-1)

Copied to clipboard

Shuming Ma, Xu Sun, Wei Li, Sujian Li, Wenjie Li, Xuancheng Ren

A Neural Question Answering Model Based on Semi-Structured Tables (C18-1)

Challenge:	Existing models tend to memorize words instead of learning meaning of words . existing models tend not to model semantic information, resulting in incorrect sentences .
Approach:	They propose a novel model that generates words by querying distributed word representations . they evaluate model on two paraphrase-oriented tasks, namely text simplification and short abstractive summarization .
Outcome:	The proposed model outperforms the baseline model on two paraphrase-oriented tasks . it achieves state-of-the-art performance on these benchmark datasets .

Copied to clipboard

Hao Wang, Xiaodong Zhang, Shuming Ma, Xu Sun, Houfeng Wang, Mengxiang Wang

Multimodality

Challenge:	Existing question answering systems rely on raw text and structured knowledge graphs.
Approach:	They build an end-to-end system to answer multiple choice questions with semi-structured tables as its knowledge.
Outcome:	The proposed system improves on the state-of-the-art question answering system with tabMCQ dataset.

BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2022.naacl-main)

Copied to clipboard

Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, Ming Zhou

A Deep Reinforced Sequence-to-Set Model for Multi-Label Classification (P19-1)

Challenge:	Standard evaluation metrics, e.g., BLEU, TER and METEOR, focus on the quality of translations at the sentence level and do not consider discourse-level features.
Approach:	They propose to use a metric to take discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans.
Outcome:	The proposed metric possesses better selectivity and interpretability at the document-level, and is more sensitive to document- level nuances.

Copied to clipboard

Pengcheng Yang, Fuli Luo, Shuming Ma, Junyang Lin, Xu Sun

Text Classification

Challenge:	Multi-label classification (MLC) aims to assign multiple labels to each sample.
Approach:	They propose a sequence-to-set model that is trained via reinforcement learning and rewards feedback independent of the label order.
Outcome:	The proposed model outperforms baseline models and reduces sensitivity to label order.

A Length-Extrapolatable Transformer (2023.acl-long)

Copied to clipboard

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei

Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data? (C18-1)

Challenge:	Existing Transformers can only deal with the in-distribution size of inputs.
Approach:	They propose a relative position embedding to explicitly maximize attention resolution . they also use blockwise causal attention during inference for better resolution a .
Outcome:	The proposed model achieves strong performance in interpolation and extrapolation settings.

Copied to clipboard

Yi Zhang, Xu Sun, Shuming Ma, Yang Yang, Xuancheng Ren

Autoencoder as Assistant Supervisor: Improving Text Representation for Chinese Social Media Text Summarization (P18-2)

Challenge:	Existing neural models take long distance dependencies into account when predicting the tag of the current token.
Approach:	They propose a method to capture long distance tag dependencies and use them for dependency analysis.
Outcome:	The proposed model can predict multiple tags for the current token without taking dependencies between tags into account.

Copied to clipboard

Shuming Ma, Xu Sun, Junyang Lin, Houfeng Wang

Phrase-level Self-Attention Networks for Universal Sentence Encoding (D18-1)

Challenge:	Existing abstractive text summarization models learn a semantic representation of the source text and the summaries from it.
Approach:	They evaluate the model on a popular Chinese social media dataset and compare it to other models.
Outcome:	The proposed model achieves state-of-the-art performance on a popular Chinese social media dataset.

Copied to clipboard

Wei Wu, Houfeng Wang, Tianyu Liu, Shuming Ma

Bitnet.cpp: Efficient Edge Inference for Ternary LLMs (2025.acl-long)

Challenge:	Phrase-level self-attention networks (PSAN) can capture context dependencies at the phrase level instead of the sentence level.
Approach:	They propose to perform self-attention across words inside a phrase to capture context dependencies at the phrase level and use the gated memory updating mechanism to refine each word’s representation hierarchically with longer-term context dependency captured in a larger phrase.
Outcome:	The proposed model can achieve state-of-the-art performance across a plethora of NLP tasks including binary and multi-class classification, natural language inference and sentence similarity.

Copied to clipboard

Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, Furu Wei

Green & Sustainable NLP

Challenge:	1-bit large language models have spurred interest in ternary LLMs, but efficient edge inference is still scarce.
Approach:	They propose an inference system optimized for 1-bit large language models . they propose a new library that facilitates sub-2-bits-per-weight inference .
Outcome:	The proposed inference system achieves 6.25x speed increase over full-precision baselines and 2.32x over low-bit baselines.

Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Network (P18-2)

Copied to clipboard

Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma

Challenge:	Existing methods to rate academic papers require a lot of feature engineering and can cause inequality.
Approach:	They propose to use a novel convolutional neural network to automatically rate academic papers . they propose to build a dataset to automatically determine whether to accept academic papers.
Outcome:	The proposed model outperforms baselines by a large margin.

How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? (2021.findings-acl)

Copied to clipboard

Weijia Xu, Shuming Ma, Dongdong Zhang, Marine Carpuat

Improving Multilingual Neural Machine Translation with Auxiliary Source Languages (2021.findings-emnlp)

Challenge:	Prior work suggests that distilled training data is less complex than manual translations.
Approach:	They propose to use sequence-level knowledge distillation to match autoregressive models' translation quality.
Outcome:	The proposed model can match translation quality of autoregressive models with distilled training data.

Copied to clipboard

Weijia Xu, Yuwei Yin, Shuming Ma, Dongdong Zhang, Haoyang Huang