An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent years have witnessed that massively multilingual neural machine translation (MMNMT) achieves a remarkable progress in both high- and low-resource language translation. |
| Approach: | They propose to use a robustness evaluation benchmark dataset to assess the translation robustness of Indonesian-Chinese translation in the face of various naturally occurring noise. |
| Outcome: | The proposed dataset is publicly available at https://github.com/ID-ZH-MTRobustEval. |
Similar Papers
Is Robustness Transferable across Languages in Multilingual Neural Machine Translation? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on bilingual machine translation with a single translation direction. |
| Approach: | They propose a robustness transfer analysis protocol to analyze the transferability of robustness across different languages in multilingual neural machine translation. |
| Outcome: | The proposed protocol shows that the robustness gained in one translation direction can transfer to other translation directions. |
Did Translation Models Get More Robust Without Anyone Even Noticing? (2025.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models are highly sensitive to “noisy” inputs, such as spelling errors, abbreviations, and formatting issues. |
| Approach: | They revisit this insight in light of recent multilingual MT models and large language models applied to machine translation. |
| Outcome: | The proposed models perform better on clean data than previous models, but none of the open models use robustness techniques. |
Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back-Translation (D19-55)
Copied to clipboard
| Challenge: | Neural Machine Translation models are sensitive to noise in the input data. |
| Approach: | They propose new methods to extend limited noisy data and further improve NMT robustness to noise while keeping the models small. |
| Outcome: | The proposed methods extend limited noisy data and improve robustness to noise while keeping the models small. |
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis (2024.findings-naacl)
Copied to clipboard
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, Lei Li
| Challenge: | Existing studies show that large language models (LLMs) can handle multilingual machine translation (MMT) However, the multilingual translation ability of LLMs remains under-explored. |
| Approach: | They evaluate eight popular LLMs including ChatGPT and GPT-4 to determine their performance in multilingual machine translation. |
| Outcome: | The proposed model can generate moderate translation even on zero-resource languages and cross-lingual exemplars can provide better task guidance for low-resourced translation than exemplar in the same language pairs. |
PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies suggest that Neural Machine Translation still struggles with certain kinds of input with considerable noise, such as User-Generated Contents (UGC) on the Internet. |
| Approach: | They propose to evaluate the robustness of Neural Machine Translation models against specific linguistic phenomena in Japanese-English translation. |
| Outcome: | The proposed model can handle user-generated content (UGC) on the Internet, but it is difficult to translate clean inputs. |
Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers (2021.acl-long)
Copied to clipboard
| Challenge: | a meta-evaluation of machine translation (MT) has been conducted in 769 research papers . a recent study shows that evaluation practices have changed over the past decade . |
| Approach: | They propose a meta-evaluation method for machine translation that uses BLEU scores to evaluate MT performance. |
| Outcome: | The proposed meta-evaluation of machine translation shows that evaluation practices have changed over the past decade . the authors suggest that the evaluation process should be streamlined and standardized to ensure the validity of the evaluation method . |
MTNT: A Testbed for Machine Translation of Noisy Text (D18-1)
Copied to clipboard
| Challenge: | Noisy input text can cause disastrous mistranslations in most modern machine translation systems. |
| Approach: | They propose a benchmark dataset for Machine Translation of Noisy Text (MTNT) they use reddit comments and professionally sourced translations to examine noise types. |
| Outcome: | The proposed dataset can provide an attractive testbed for noise-robust machine translation systems. |
Evaluating Robustness to Input Perturbations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that Neural Machine Translation models are brittle to small perturbations in the input. |
| Approach: | They propose to use subword regularization to measure the relative degradation and changes in translation when perturbations are added to the input. |
| Outcome: | The proposed measures show that the models are more robust to perturbations when subword regularization methods are used. |
Ask Language Model to Clean Your Noisy Translation Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models exhibit a noticeable decline in translation quality when exposed to noisy input. |
| Approach: | They use a dataset to evaluate the robustness of NMT models against noisy inputs. |
| Outcome: | The proposed dataset cleaners the noise from the target sentences while preserving the semantic integrity of the original sentences. |